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Preface 


This  report  is  the  result  of  a  collaborative  effort  between  the  National  Defense  Industrial  Associa¬ 
tion  (NDIA)  Systems  Engineering  Effectiveness  Committee  (SEEC)  and  the  Software  Engineer¬ 
ing  Institute  (SEI)  of  Carnegie  Mellon  University.  It  is  the  result  of  over  three  years  of  effort. 

While  the  output  of  this  survey  activity  is  complete  in  its  current  form,  and  needs  no  further  work 
to  find  application  in  the  defense  industry  today,  it  also  suggests  some  directions  for  future  activi¬ 
ties.  More  research  is  needed.  This  work  is  merely  a  first  step  in  a  continuing  effort  to  understand 
and  measure  the  impacts  of  Systems  Engineering. 

It  should  be  emphasized  that  the  analysis  results  and  graphs  described  throughout  this  report  de¬ 
pend  fully  on  the  mapping  of  survey  questions  to  associated  analysis  groupings.  When  interpret¬ 
ing  analysis  findings,  readers  are  strongly  encouraged  to  refer  to  Section  5  of  this  report  where 
this  mapping  is  defined  so  the  analyses  can  be  considered  in  appropriate  context.  Rather  than  rely¬ 
ing  on  vague  definitions  or  impressions  of  systems  engineering  and  the  activities  that  comprise  it, 
for  which  there  is  no  clearly  defined  consensus  across  industry,  from  the  perspective  of  this  sur¬ 
vey  these  components  are  defined  by  said  mapping,  based  generally  on  a  well-recognized  refer¬ 
ence  standard  (CMMI®). 

Note  that  this  mapping  of  responses  to  analysis  areas  is  itself  subject  to  interpretation  or  debate 
(and  is  indeed  a  continued  topic  of  discussion  even  within  the  SEEC).  Different  mappings  could, 
to  some  degree,  naturally  produce  different  analyses  and  findings.  To  maximize  the  likelihood  of 
participant  responses  to  the  survey,  the  question  set  itself  was  prioritized  and  shortened,  with  the 
result  that  individual  analysis  areas  are  addressed  at  varying  levels  of  detail. 

The  SEEC  is  aware  of  only  one  clearly  erroneous  mapping  (in  the  Validation  SE  capability),  dis¬ 
covered  late  in  the  editing,  review,  and  publication  process  for  this  report.  After  some  assessment, 
the  impact  was  not  judged  to  be  significant  on  the  resulting  analyses  or  conclusions,  but  the  re¬ 
work  of  graphs  and  text  would  have  been  extensive  enough  to  delay  promised  schedules  for  deliv¬ 
ery  of  this  report  to  survey  participants  and  other  stakeholders — this  was  determined  to  be  a 
higher  priority. 

Summarized  simply,  the  questions,  mappings  and  analyses  (imperfect  or  not)  help  establish  an 
initial  baseline  for  quantifying  the  effectiveness  of  systems  engineering  and  the  associated  impact 
on  program  performance.  It  is  hoped  these,  too,  will  be  part  of  an  ongoing  dialog  within  the  sys¬ 
tems  engineering  community  for  follow-on  work. 
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Executive  Summary 


The  Systems  Engineering  Division  (SED)  of  the  National  Defense  Industrial  Association  (NDIA) 
established  the  Systems  Engineering  Effectiveness  Committee  (SEEC)  to  obtain  quantitative  evi¬ 
dence  of  the  effect  of  Systems  Engineering  (SE)  best  practices  on  Project  Performance.  The  SEEC 
developed  and  executed  a  survey  of  defense  industrial  contractors  (i.e.,  suppliers  to  the  govern¬ 
ment)  to  identify  the  SE  best  practices  utilized  on  defense  projects,  collect  performance  data  on 
these  projects,  and  search  for  relationships  between  the  application  of  these  SE  best  practices  and 
Project  Performance. 

The  SEEC  surveyed  a  sample  of  the  population  of  major  government  contractors  and  subcontrac¬ 
tors  consisting  of  contractors  and  subcontractors  represented  in  the  NDIA  SED. 

The  survey  questionnaire  was  developed  using  the  SE  expertise  and  the  broad  diversity  of  experi¬ 
ence  of  the  SEEC  members.  The  questionnaire  consisted  of  three  sections;  one  to  identify  the 
characteristics  of  the  responding  project,  a  second  to  assess  the  project’s  utilization  of  SE  best 
practices,  and  a  third  to  collect  measures  of  Project  Performance. 

The  survey  data  was  collected  by  the  Carnegie  Mellon  ”  Software  Engineering  Institute  (SEI)  via 
the  Web.  Policies  ensuring  the  anonymity  of  the  respondents  and  the  confidentiality  of  their  re¬ 
sponses  were  enforced  to  protect  the  competition-sensitive  information  supplied.  Responses  suffi¬ 
cient  for  most  analyses  were  received  from  a  total  of  46  projects;  another  18  projects  provided 
partial  responses  useful  for  basic  descriptive  purposes.  These  responses  were  analyzed  by  the  SEI 
to  identify  relationships  between  the  deployment  of  SE  best  practices  and  overall  project/program 
performance.  The  results  of  this  analysis  are  published  in  this  report.  Only  aggregated  results  are 
contained  in  this  report;  no  information  traceable  to  any  individual  respondent,  project,  or  organi¬ 
zation  is  included. 

The  questionnaire  was  designed  to  assess  the  project’s  Systems  Engineering  Capability  (SEC)  as 
measured  by  its  utilization  of  SE  best  practices.  Project  Performance  was  then  assessed  based  on 
satisfaction  of  project  cost,  schedule,  and  scope  goals.  The  analysis  consisted  of 

•  Processing  the  respondent’s  answers  to  compute  a  score  for  that  project’s  SEC 

•  Numerically  ordering  the  SEC  scores  and  separating  them  into  three  approximately  equally 

2 

sized  groups  labeled  “Lower  Capability,”  “Moderate  Capability,”  and  “Higher  Capability” 

•  Processing  the  respondent’s  answers  to  compute  a  score  for  that  project’s  performance  ( Perf) 

•  Numerically  ordering  the  Perf  scores  and  separating  them  into  three  approximately  equally 

sized  groups  labeled  “Lower  Performance”,  “Moderate  Performance,”  and  “Best  Perform- 

„2 

ance 

•  Measuring  the  strength  of  the  relationship  between  the  Capability  and  Performance  scores. 

®  Carnegie  Mellon  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 

2  Note  that  the  terms  “Lower,”  “Moderate,”  and  “Higher”  are  relative  terms  placing  each  SE  Capability  score  or  each 
Performance  score  approximately  within  the  lower,  middle,  or  upper  third  of  the  range  of  received  responses. 
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This  analysis,  as  seen  in  Figure  1,  showed  that  projects  with  better  Systems  Engineering  Capabili¬ 
ties  delivered  better  Project  Performance. 


Project  Performance  vs.  Systems  Engineering  Capability 
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Figure  1:  Project  Performance  Versus  Systems  Engineering  Capability 

To  better  understand  the  relationship  between  SE  Capability  and  Project  Performance,  the  ques¬ 
tionnaire’s  assessment  of  SE  Capability  looked  at  12  areas  of  SE  Capability,  addressing  the  pro¬ 
ject’s  utilization  of  SE  best  practices  in  each  area.  Further  details  regarding  the  contents  of  these 
process  areas  are  described  in  the  body  of  this  Special  Report.  As  with  the  relationship  between 
total  SE  Capability  and  Performance,  the  responses  were  analyzed  to  identify  relationships  be¬ 
tween  Project  Performance  and  the  project’s  use  of  best  practices  in  each  of  the  process  areas. 
Table  1  summarizes  these  relationships. 
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Table  1:  Summary  of  Project  Performance  Versus  Systems  Engineering  Capability 


Supplier’s  Systems  Engineering 

Capability3 

Relationship  to 

Project  Performance 

Relationship 

(Gamma4) 

Section 

Reference 

Project  Planning 

Weak  positive  relationship 

+0.13 

5.1. 3.2 

Project  Monitoring  and  Control 

Weak  negative  relationship 

-0.13 

5.1. 3. 3 

Risk  Management 

Moderately  strong  positive  relation¬ 
ship 

+0.28 

5.1. 3.4 

Requirements  Development  and  Management 

Moderately  strong  positive  relation¬ 
ship 

+0.33 

5.1. 3. 5 

Trade  Studies 

Moderately  strong  positive  relation¬ 
ship 

+0.37 

5.1. 3.6 

Product  Architecture 

Moderately  strong  to  strong  positive 
relationship 

+0.40 

5.1. 3. 7 

Technical  Solution 

Moderately  strong  positive  relation¬ 
ship 

+0.36 

5.1. 3.8 

Product  Integration 

Weak  positive  relationship 

+0.21 

5. 1.3. 9 

Verification 

Moderately  strong  positive  relation¬ 
ship 

+0.25 

5.1.3.10 

Validation 

Moderately  strong  positive  relation¬ 
ship 

+0.28 

5.1.3.11 

Configuration  Management 

Weak  positive  relationship 

+0.13 

5.1.3.12 

IPT-Related  Capability 

Moderately  strong  positive  relation¬ 
ship 

+0.34 

5.1. 3.1 

Additionally,  the  survey  examined  the  relationship  between  Project  Challenge  and  Project  Per¬ 
formance.  Project  Challenge  was  measured  by  factors  such  as  included  life-cycle  phases,  sources 
of  technical  challenge,  total  project  effort,  inter-organizational  complexity,  contract  value,  etc. 
Table  2  summarizes  the  relationships  for  each  area. 


Table  2:  Summary  of  Project  Performance  Versus  Project  Challenge 


Project  Challenge  Factor 

Relationship  to 

Project  Performance 

Relationship 

(Gamma) 

Section 

Reference 

Project  Challenge 

Moderately  strong  negative  relation¬ 
ship 

-0.31 

5.1.1 

3  Use  caution  to  avoid  over-interpreting  the  meaning  of  the  Systems  Engineering  Capability  (SEC)  and  Project 
Challenge  categories  listed  in  Table  1  through  Table  3.  For  example,  the  “Project  Planning”  category  does  include 
elements  of  project  planning,  but  is  not  a  comprehensive  compilation  of  all  project  planning  activities.  To  properly 
understand  the  listed  relationships,  please  refer  to  the  report  sections  listed  in  the  last  column  to  better  under¬ 
stand  the  contents  of  each  category. 

4  Gamma  is  a  measure  of  association  that  expresses  the  strength  of  relationship  between  two  ordinal  variables, 
with  values  near  -1  indicating  a  strong  opposing  relationship,  values  near  0  indicating  a  weak  or  no  relationship 
(statistical  independence),  and  values  near  +1  indicating  a  strong  supporting  relationship 
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The  survey  also  examined  Project  Environment  factors  that  may  or  may  not  influence  Project  Per¬ 
formance.  Due  to  the  relatively  small  sample  size  and  the  small  number  of  respondents,  the  num¬ 
ber  of  projects  in  each  answer  category  for  the  Project  Environment  questions  was  sufficiently 
small  to  reduce  the  confidence  one  can  have  in  these  findings.  Results  are  presented  in  this  report, 
but  care  should  be  taken  not  to  over-interpret  these  differences. 

Finally,  the  survey  examined  the  impact  on  Project  Performance  of  the  capabilities  of  the  organi¬ 
zation  acquiring  the  project  (i.e.,  the  organization  issuing  and  managing  the  contract  to  the  sup¬ 
plier).  Although  the  survey  was  not  specifically  designed  to  provide  a  detailed  assessment  of  these 
Acquirer  Capabilities,  some  responses  from  the  suppliers  could  be  used  to  develop  a  rudimentary 
relative  measure  of  some  acquirer  capabilities.  The  scope  of  the  acquirer  assessment  consisted  of 
only  a  few  questions.  Due  to  this  narrow  scope,  and  due  to  the  indirect  nature  of  this  assessment 
(i.e.,  assessment  of  acquirers  via  responses  from  suppliers),  this  survey  was  unable  to  identify 
clear  relationships  between  Acquirer  Capability  and  Project  Performance. 

The  moderately  strong  statistical  relationships  between  Systems  Engineering  Capabilities  and 
Project  Performance  shown  earlier  in  this  Executive  Summary  are  notable  by  themselves.  Elow- 
ever,  notably  stronger  relationships  are  apparent  by  combining  the  effects  of  more  than  one  of  the 
best  practices  categories,  as  shown  in  Table  3. 


Table  3:  Project  Performance  Versus  aggregated  Systems  Engineering  Capabilities 


Supplier  Systems  Engineering  Capability 

Relationship  to 

Project  Performance 

Relationship 

(Gamma) 

Section 

Reference 

Total  Systems  Engineering  Capability 

Moderately  strong  positive  relation¬ 
ship 

+0.32 

5.1.3.13 

Combined  Requirements  and 

Technical  Solution  Capability 

Strong  positive  relationship 

+0.49 

5.2.3.14 

Requirements  and  Technical 

Solution  Combined  with  Project  Challenge 

Very  strong  positive 

+0.63 

5.3.1. 3 

Of  course,  Systems  Engineering  Capability  alone  does  not  ensure  outstanding  Project  Perform¬ 
ance.  The  survey  results  show  notable  differences  in  the  relationship  between  SE  best  practices 
and  performance  among  more  challenging  as  compared  to  less  challenging  projects  (section 
5.3.1).  The  statistical  relationship  with  Project  Performance  is  quite  strong  for  survey  data  of  this 
kind  when  both  SE  Capability  and  Project  Challenge  are  considered  together  (Gamma  =  0.63; 
section  5. 1.3. 3). 

This  relationship  is  illustrated  in  Figure  2. 
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Figure  2:  Performance  vs.  Project  Challenge  and  Overall  SE  Capability 
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Abstract 


This  survey  quantifies  the  relationship  between  the  application  of  Systems  Engineering  (SE)  best 
practices  to  projects  and  programs,  and  the  performance  of  those  projects  and  programs.  The  survey 
population  consisted  of  projects  and  programs  executed  by  defense  contractors  who  are  members  of 
the  Systems  Engineering  Division  (SED)  of  the  National  Defense  Industrial  Association  (NDIA). 
The  deployment  of  SE  practices  on  a  project  or  program  was  measured  through  the  availability  and 
characteristics  of  specific  SE-related  work  products.  Project  Performance  was  measured  through 
typically  available  project  measures  of  cost  performance,  schedule  performance,  and  scope  perform¬ 
ance.  Additional  project  and  program  information  such  as  project  size,  project  domain,  and  other 
data  was  also  collected  to  aid  in  characterizing  the  respondent’s  project.  Analysis  of  the  survey  re¬ 
sponses  revealed  moderately  strong  statistical  relationships  between  Project  Performance  and  sev¬ 
eral  categorizations  of  specific  of  SE  best  practices.  Notably  stronger  relationships  are  apparent 
by  combining  the  effects  of  more  than  one  the  best  practices  categories.  Of  course,  Systems  Engi¬ 
neering  Capability  alone  does  not  ensure  outstanding  Project  Performance.  The  survey  results 
show  notable  differences  in  the  relationship  between  SE  best  practices  and  performance  between 
more  challenging  as  compared  to  less  challenging  projects.  The  statistical  relationship  between 
Project  Performance  and  the  combination  of  SE  Capability  and  Project  Challenge  is  quite  strong 
for  survey  data  of  this  type. 
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1  Introduction 


The  mission  of  the  National  Defense  Industrial  Association  (NDIA)  Systems  Engineering  Divi¬ 
sion  (SED)  is  to  promote  the  widespread  use  of  Systems  Engineering  in  the  government  acquisi¬ 
tion  process  in  order  to  achieve  affordable  and  supportable  systems  that  meet  the  needs  of  defense 
agency  and  civil  agency  users.  [NDIA  2007]  In  pursuit  of  this  mission,  the  NDIA  SED  tasked  the 
Systems  Engineering  Effectiveness  Committee  (SEEC)  to  research  and  report  on  the  costs  and 
benefits  associated  with  Systems  Engineering  practices  in  the  acquisition  and  development  of  de¬ 
fense  and  civil  agency  systems. 


1.1  BACKGROUND 

Case  studies  and  surveys  are  among  the  various  methods  used  to  assess  the  effectiveness  and  im¬ 
pact  of  actions  and  processes.  Both  are  useful  tools,  each  complementing  the  other. 

Case  studies  provide  an  in-depth  analysis  of  one  (or  a  few)  specific  case(s).  This  analysis  can  pro¬ 
vide  insight  into  causality  (for  example,  action  A  caused  benefit  B).  While  a  case  study  may  be 
persuasive  in  its  presentation  and  analysis  of  information,  it  remains  anecdotal  in  nature.  Because 
it  evaluates  only  one  (or  a  few)  specific  case(s),  readers  may  dispute  the  applicability  of  the  find¬ 
ings  to  their  circumstances  and  their  organizations.  Furthermore,  the  degree  to  which  the  findings 
may  be  applied  and/or  extrapolated  to  different  circumstances  and  different  organizations  may  not 
be  known. 

Surveys  provide  a  less  comprehensive  analysis  of  a  larger  number  of  cases  and  can  be  highly  use¬ 
ful  for  showing  statistical  relationships  (wherever  action  A  is  taken,  benefit  B  is  likely  to  be 
found).  The  results  of  the  surveys  are  statistical  in  nature,  rather  than  anecdotal,  and  their  findings 
are  usually  more  generalizable  and  applicable  to  the  wider  domain  of  the  survey  population.  Many 
surveys  are  self-administered  (that  is,  the  respondent  reads  the  survey  questionnaire,  and  gener¬ 
ates  a  response  based  upon  his  or  her  understanding  of  the  question).  In  such  cases,  the  surveyor 
must  strive  to  for  clarity  in  the  survey  questions,  since  he  or  she  has  no  opportunity  to  verify 
and/or  correct  the  respondent’s  interpretations. 


1.2  PURPOSE 

Case  studies  and  anecdotal  reports  have  shown  that  properly  implemented  systems  engineering 
can  yield  significant  benefits  for  a  project.  And  yet,  broadly  applicable  quantification  of  these 
costs  and  benefits  remains  elusive.  This  was  the  impetus  for  the  formation  of  the  SEEC — to  an¬ 
swer  the  questions 

1 .  What  will  the  application  of  Systems  Engineering  practices  cost  me? 

2.  What  benefits  will  I  gain  from  the  application  of  these  practices? 
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While  one  would  expect  an  organization  with  accurate  project  cost  accounting  methods  to  be  able 
to  identify  the  cost  of  efforts  dedicated  to  Systems  Engineering,  this  is  not  always  the  case.  For 
many  projects,  Systems  Engineering  is  not  an  identified,  segregated  effort  with  a  dedicated  budg¬ 
et.  Often,  Systems  Engineering  effort  is  distributed  across  many  project  tasks  and  is  planned  not 
independently,  but  as  an  element  of  those  tasks.  As  such,  it  may  be  difficult  to  know  both  what  the 
original  budget  was  for  Systems  Engineering,  and  what  actual  Systems  Engineering  expenditures 
have  been.  Furthermore,  a  commonly  accepted  definition  of  Systems  Engineering  does  not  exist. 
As  such,  activities  that  would  be  considered  Systems  Engineering  in  one  organization  may  be 
considered  as  project  management  or  something  else  in  another  organization.  Thus,  even  if  data 
on  Systems  Engineering  activities  is  available,  comparison  of  such  data  across  multiple  organiza¬ 
tions  is  not  possible. 

Quantifying  the  answer  to  the  second  question  is  even  more  difficult,  since  the  benefits  derived 
from  effective  Systems  Engineering  may  be  less  obvious  and  less  tangible.  Some  of  the  benefits 
take  the  form  of  cost  avoidance  (for  example,  avoiding  rework  arising  from  interface  mis¬ 
matches).  Some  take  the  form  of  improved  efficiency  (such  as  defining  product  and  organiza¬ 
tional  structures  that  promote  effective  division  of  work).  Some  take  the  form  of  improved  prod¬ 
uct  performance  (for  example,  better  understanding  and  satisfaction  of  user  needs  and  key 
performance  parameters). 

Because  the  cost  of  Systems  Engineering  effort  is  not  explicitly  planned  and  the  benefits  are  not 
readily  known,  the  case  for  the  dedication  of  resources  to  Systems  Engineering  activities  can  be 
difficult  to  make.  In  fact,  some  projects  are  tempted  to  reduce  the  amount  of  Systems  Engineering 
applied  as  a  means  of  reducing  schedule  and  cost.  This  reduction  may  take  the  form  of 

•  reduction  (or  elimination)  of  Systems  Engineering  efforts  within  the  acquiring  Program  Of¬ 
fice 

•  pressure  on  the  contractor  from  the  acquiring  Program  Office  to  reduce  Systems  Engineering 
expenditures  to  limit  contract  cost 

•  pressure  from  the  contractor’s  management  to  reduce  Systems  Engineering  expenditures  to 
reduce  the  bid  price. 

The  intent  of  this  survey  was  to  identify  the  impact  of  Systems  Engineering  efforts  by  sampling 
projects  at  a  number  of  development  contractors  to  identify  the  degree  of  statistical  relationship 
between  the  use  of  SE  best  practices  applied  to  a  project  and  the  performance  of  that  project. 


1.3  SURVEY  HYPOTHESIS 

A  basic  tenet  of  statistical  studies  is  to  establish  an  hypothesis  and  then  test  for  the  validity  of  that 
hypothesis.  In  this  particular  case,  we  are  asserting  that  the  performance  of  SE  best  practices  has  a 
measurable,  positive  impact  on  program  execution,  as  stated  below. 
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HYPOTHESIS 


The  effective  performance  of  SE  best  practices  on  a  development  program  yields  quantifiable  im¬ 
provements  in  the  program  execution  (e.g.,  improved  cost  performance,  schedule  performance, 

technical  performance). 


The  alternative  to  this  hypothesis  (often  referred  to  as  the  null  hypothesis)  is  that  the  perfonnance 
of  SE  best  practices  has  no  effect  (or  a  negative  effect)  on  program  performance.  The  goal  of  our 
survey  is  to  collect  and  analyze  data  to  choose  between  these  two  hypotheses.  In  theory,  this  could 
be  accomplished  by 

1 .  identifying  a  number  of  programs  that  utilize  SE  best  practices,  and  collecting  their  program 
performance  measures 

2.  identifying  a  number  of  programs  that  do  not  utilize  SE  best  practices,  and  collecting  their 
program  performance  measures 

3.  comparing  the  two  sets  of  program  performance  measures  to  identify  statistically  significant 
differences,  if  any 

In  reality,  the  process  is  complicated  by  the  following  issues: 

•  We  have  no  reliable  way  of  identifying  programs  that  do  or  do  not  use  SE  best  practices. 

•  Program  performance  measures  are  crude  measures  of  actual  program  performance. 

•  Program  perfonnance  measures  are  influenced  by  factors  other  than  SE  activities  (e.g.,  re¬ 
quirements  stability,  technical  challenge,  and  other  factors). 

To  address  the  first  of  these  bulleted  issues,  we  crafted  a  survey  that  not  only  captures  program 
performance  measures,  but  also  assesses  the  use  of  SE  best  practices  in  a  quantifiable  manner. 

The  use  of  SE  best  practices  by  contractors  varies  over  a  continuum  from  those  that  do  not  use 
best  practices  to  those  that  use  best  practices  extensively.  By  collecting  data  to  enable  assessment 
of  SE  best  practice  utilization  across  this  continuum,  we  can  look  for  relationships  between  SE 
best  practice  usage  and  program  performance. 

We  address  the  second  of  these  issues  by  collecting  multiple  performance  measures  (such  as 
EVMS  data,  milestone  satisfaction  data,  and  others)  and  looking  for  the  degree  of  agreement  be¬ 
tween  these  measures. 

We  address  the  third  of  these  issues  through  the  assertion  that  many  of  the  other  factors  that  influ¬ 
ence  program  performance  are  uncorrelated  with  the  use  of  SE  best  practices.  For  example,  there 
is  no  reason  to  believe  that  contractors  that  use  SE  best  practices  are  blessed  with  programs  hav¬ 
ing  contracted  requirements  of  better  quality  than  are  contractors  that  do  not  use  SE  best  practices. 
This  assertion  is  also  tested  in  the  survey  by  collecting  measures  of  some  of  these  other  factors, 
enabling  the  evaluation  of  the  asserted  orthogonality. 
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2  Survey  Development 


2.1  STEP  1 :  DEFINE  THE  GOAL 

The  role  of  the  SEEC  is  defined  on  the  NDIA  Web  site  as  follows: 

[The  Systems  Engineering  Effectiveness]  Subcommittee  attempts  to 
identify  those  key  critical  skills  and  tools  that  are  essential  for  implementa¬ 
tion  of  a  robust  Systems  Engineering  process.  It  works  to  identify  success- 
oriented  approaches  to  systems  engineering,  and  help  promote  these  con¬ 
cepts  throughout  industry  and  the  department  of  defense.  ...” 

The  identification  of  critical  skills,  tools,  and  success-oriented  approaches  will  not  aid  projects  if 
they  do  not  use  them;  and  they  will  not  use  them  unless  they  are  convinced  that  their  benefits  ex¬ 
ceed  their  cost.  Thus,  the  goal  of  this  survey  was: 

Goal:  Identify  the  degree  of  statistical  association  between  the  use  of  spe¬ 
cific  Systems  Engineering  practices  and  activities  on  projects,  and 
quantitative  measures  of  Project  Performance. 

2.2  STEP  2:  CHOOSE  THE  SURVEYED  POPULATION 

The  second  step  was  to  choose  the  population  to  be  included  in  the  survey.  As  this  survey  activity 
was  sponsored  by  the  NDIA,  the  SEEC  elected  to  focus  primarily  on  projects  involving  defense 
and  other  government  agencies.  Thus,  candidate  groups  for  inclusion  in  the  survey  included 

•  government  program  offices  (civil  and  defense  agencies) 

•  major  government  contractors 

•  subcontractors  to  major  government  contractors. 

The  parameters  of  the  study  could  vary  considerably  based  upon  the  inclusion  or  exclusion  of 
each  of  these  groups.  Additionally,  a  means  of  sampling  within  these  groups  was  also  needed. 

The  consensus  of  the  SEEC  was  that,  among  these  groups,  the  impact  of  SE  would  be  greater 
among  the  contractors  and  subcontractors  than  at  the  program  offices.  Furthermore,  we  believed 
that  data  availability  would  be  higher  in  the  contractor  and  subcontractor  groups.  Thus,  the  SEEC 
chose  to  direct  this  survey  at  a  population  consisting  of  major  government  contractors  and  sub¬ 
contractors.  Although  this  population  is  quite  large,  consisting  of  thousands  of  suppliers,  the 
member  companies  of  the  NDIA  Systems  Engineering  Division  (SED)  are  a  representative  subset 
of  this  population.  The  NDIA  SED  maintains  a  roster  of  the  485  “active”  members  (that  is,  those 
who  have  recently  attended  NDIA  SED  meetings).  After  filtering  this  list  for  organizations  that 
supply  products  (as  opposed  to  services)  to  defense  and  government  acquirers,  the  SEEC  pro¬ 
duced  a  list  of  50  companies  to  invite  to  participate  in  the  survey.  The  intent  of  the  survey  was  to 
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collect  data  at  the  project  level  (rather  than  at  the  organizational  level);  thus,  each  of  these  50  or¬ 
ganizations  could  contribute  multiple  projects  to  participate  in  the  survey. 

Surveys  of  other  populations  (i.e.,  government  program  offices)  may  be  conducted  in  the  future,  if 
warranted. 

2.3  STEP  3:  DEFINE  THE  MEANS  TO  ASSESS  USAGE  OF  SE  PRACTICES 

The  third  step  was  to  define  the  methods  used  to  assess  the  application  of  SE  practices  to  projects. 
While  various  SE  models,  standards,  and  so  forth  can  inform  this  decision  (such  as  CMMI- 
SE/SW,  EIA  632,  MIL-STD-499B,  IEEE-STD-1220,  ISO/IEC- 15288,  and  others),  this  effort  was 
hampered  by  the  fact  that  a  widely  accepted  definition  of  what  constitutes  SE  does  not  exist.  To 
overcome  this  obstacle,  the  SEEC  chose  to  survey  specific  activities  that  would  normally  be  re¬ 
garded  as  elements  of  SE.  The  survey  analysis  then  examines  the  relationships  between  these  ac¬ 
tivities  and  overall  Project  Performance.  Thus,  for  any  activity  that  did  not  fit  a  particular  reader’s 
preferred  definition  of  SE,  the  analysis  results  for  that  activity  could  be  ignored.  In  general,  the 
focus  of  the  SE  practice  assessment  was  placed  on  identifying  tangible  artifacts  of  SE  activities. 

The  SEEC  chose  to  base  this  assessment  primarily  upon  the  Capability  Maturity  Model  Integra¬ 
tion  (CMMI)  due  to  the  SED’s  sponsorship  of  this  model,  as  well  as  the  SEEC’s  familiarity  with 
it.  Starting  with  the  CMMI-SE/SW/IPPD  Model  v  1.1,  we  identified  the  work  products  that,  in  the 
judgment  of  the  SE  experts  on  the  committee,  result  from  Systems  Engineering  tasks.  The  pres¬ 
ence  of  these  work  products  provides  an  indication  of  the  magnitude  of  the  Systems  Engineering 
activities  performed  on  the  project.  Questions  were  worded  to  search  for  the  content  of  these  sug¬ 
gested  work  products,  rather  than  the  specified  work  products  themselves,  thereby  enabling  the 
reporting  project  to  accurately  represent  their  system  engineering  activities,  regardless  of  the  titles 
or  format  of  their  specific  work  products. 

This  approach  enabled  us  to  analyze  relationships  between  Project  Performance  and  Systems  En¬ 
gineering  work  products  both  individually  and  in  ensemble,  searching  for  those  work  products 
most  closely  tied  to  project  success. 

The  process  of  identifying  Systems  Engineering  work  products  was  as  follows: 

1 .  Extract  all  listed  work  products  from  the  CMMI. 

The  CMMI  SW/SE/IPPD  vl.l  consists  of  614  practices  needed  to  satisfy  179  goals  organ¬ 
ized  into  25  process  areas.  The  model  also  lists  476  typical  work  products  produced  by  these 
practices.  While  this  list  of  work  products  is  not  all-inclusive,  it  provides  a  reasonable 
framework  that  can  be  used  to  organize  a  search  for  Systems  Engineering  artifacts. 

2.  Identify  work  products  that  (in  the  judgment  of  the  SEEC)  result  from  Systems  Engi¬ 
neering  activities. 

Filter  these  work  products  to  extract  those  that  are  (in  the  judgment  of  the  SEEC  SE  experts) 
the  result  of  activities  that  would  normally  be  classified  as  Systems  Engineering.  Developing 
a  firm  definition  of  what  is  and  what  is  not  Systems  Engineering  is  not  critical  to  this  proc¬ 
ess.  By  looking  for  defined  work  products  resulting  from  defined  practices,  we  eliminate  the 
subjectivity  of  a  Systems  Engineering  definition.  At  the  end  of  the  analysis  phase  of  the  sur- 
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vey,  we  will  have  related  Project  Performance  with  these  defined  work  products  and  prac¬ 
tices.  If  we  choose  to  define  Systems  Engineering  as  encompassing  these  practices  and  work 
products,  then  we  can  relate  Project  Performance  to  this  definition  of  Systems  Engineering. 
In  the  event  that  a  particular  reader  of  this  analysis  disagrees  with  that  definition  of  Systems 
Engineering,  it  will  still  be  possible  for  them  to  examine  the  relationship  between  Project 
Performance  and  the  defined  practices  and  work  products. 

As  a  result  of  this  filtering  process,  the  SEEC  has  identified  a  subset  of  87  practices  needed 
to  satisfy  3 1  goals  organized  into  14  process  areas.  These  practices  produce  199  work  prod¬ 
ucts. 

3.  Extract  those  work  products  that  are  (in  the  judgment  of  the  SEEC)  most  significant. 

In  a  survey  such  as  this,  one  must  be  concerned  with  the  demands  placed  upon  the  potential 
respondents.  If  they  are  asked  for  information  that  is  not  readily  available,  or  are  expected  to 
spend  a  significant  amount  of  time  to  complete  the  questionnaire,  the  response  rate  may  drop 
precipitously.  For  this  reason,  it  is  not  practical  to  address  all  185  work  products  identified  in 
the  previous  process.  To  shorten  the  questionnaire,  it  is  necessary  to  address  only  the  most 
significant  of  these  work  products.  Significance  is  defined  as 

•  those  work  products  that  are  thought  (in  the  judgment  of  the  SEEC  SE  experts)  to  have 

the  greatest  impact  on  the  project 

•  those  work  products  that  are  thought  (in  the  judgment  of  the  SEEC  SE  experts)  to  have 

the  greatest  ability  to  discriminate  between  projects  that  have  effective  Systems  Engi¬ 
neering,  and  those  that  do  not 

As  a  result  of  this  filtering  process,  the  SEEC  has  identified  a  subset  of  45  practices  needed 
to  satisfy  23  goals  organized  into  13  process  areas.  These  practices  produce  71  work  prod¬ 
ucts. 

This  process  is  illustrated  in  Figure  3;  a  summary  of  the  results  are  found  in  Table  4;  and  the  de¬ 
tails  of  the  process  and  its  outcome  are  found  in  Table  4. 
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CMMI-SW/SE  vi.i 

25  Process  Areas 
179  Goals 
614  Practices 
476  Work  Products 


Systems 
Engineering- 
related  Filter 


Size  Constraint 
Filter 


Considered  significant 
to  Systems  Engineering 


14  Process  Areas 
31  Goals 
87  Practices 
199  Work  Products 


X 


13  Process  Areas 
23  Goals 
45  Practices 
71  Work  Products  M 


Figure  3:  SE  Characterization  Process 
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PROCESS  AREAS 

CMMI  SW/SE  Total 

SE  Related 

Significant  SE  J 

Specific 

Generic 

Specific 

Generic 

Specific 

Generic  [ 

Goals 

Practices 

Work 

Products 

Goals 

Practices 

Work 

Products _ 

Goals 

Practices 

“Work 

Products 

Goals 

Practices 

Work 

Products _ 

Goals 

Practices 

“Work 

Products _ 

Goals 

Practices 

Work 

Products _ 

PROCESS  MANAGEMENT 

Organizational  Process  Focus 

2 

7 

14 

5 

17 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Organizational  Process  Definition 

1 

5 

11 

5 

17 

0 

1 

1 

1 

0 

0 

0 

1 

1 

1 

0 

0 

0 

Organizational  Training 

2 

7 

13 

5 

17 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Organizational  Process  Perform¬ 
ance 

1 

5 

5 

5 

17 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Organizational  Innovation  and  De¬ 
ployment 

2 

7 

11 

5 

17 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

PROJECT  MANAGEMENT 

Project  Planning 

3 

14 

52 

5 

17 

1 

2 

7 

22 

1 

1 

1 

2 

3 

9 

1 

1 

1 

Project  Monitoring  and  Control 

2 

10 

11 

5 

17 

0 

2 

7 

0 

0 

0 

0 

2 

6 

0 

0 

0 

0 

Supplier  Agreement  Management 

2 

7 

26 

5 

17 

0 

1 

1 

5 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Integrated  Project  Management 

4 

13 

46 

5 

17 

0 

1 

3 

14 

0 

0 

0 

1 

2 

3 

0 

0 

0 

Risk  Management 

3 

7 

16 

5 

17 

0 

2 

3 

6 

0 

0 

0 

2 

3 

6 

0 

0 

0 

Integrated  Teaming 

2 

8 

25 

5 

17 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Integrated  Supplier  Management 

2 

5 

16 

5 

17 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Quantitative  Project  Management 

2 

8 

23 

5 

17 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

ENGINEERING 

Requirements  Management 

1 

5 

13 

5 

17 

1 

1 

5 

13 

1 

1 

1 

1 

4 

9 

0 

1 

1 

Requirements  Development 

3 

12 

28 

5 

17 

0 

3 

10 

28 

0 

0 

0 

3 

4 

8 

0 

0 

0 

Technical  Solution 

3 

11 

30 

5 

17 

1 

3 

11 

30 

1 

1 

1 

2 

7 

12 

1 

1 

1 

Product  Integration 

3 

9 

27 

5 

17 

0 

2 

5 

16 

0 

0 

0 

1 

1 

1 

0 

0 

0 

Verification 

2 

8 

24 

5 

17 

0 

2 

9 

20 

0 

0 

0 

2 

5 

10 

0 

0 

0 

Validation 

2 

5 

16 

5 

17 

0 

2 

4 

11 

0 

0 

0 

1 

1 

2 

0 

0 

0 

SUPPORT 

Configuration  Management 

3 

7 

16 

5 

17 

0 

3 

7 

16 

0 

0 

0 

3 

5 

7 

0 

0 

0 

Process  and  Product  QA 

2 

4 

13 

5 

17 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Measurement  and  Analysis 

2 

8 

12 

5 

17 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Decision  Analysis  and  Resolution 

1 

6 

7 

5 

17 

0 

1 

6 

7 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Org’l  Environment  for  Integration 

2 

6 

15 

5 

17 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Causal  Analysis  and  Resolution 

2 

5 

6 

5 

17 

0 

2 

5 

7 

0 

0 

0 

0 

0 

0 

0 

0 

0 

TOTAL 

54 

189 

476 

125 

425 

3 

28 

84 

196 

3 

3 

3 

21 

42 

68 

2 

3 

3 

Table  4:  Systems  Engineering  Work  Product  Selection 
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2.4  STEP  4:  DEFINE  THE  MEASURED  BENEFITS  TO  BE  STUDIED 


The  object  of  this  study  is  to  provide  quantitative  assessment  of  the  value  of  SE  practices.  To  ac¬ 
complish  this,  we  need  quantitative  measures  of  Project  Performance.  In  order  to  maximize  the 
availability  of  data  from  the  participants,  we  utilized  measures  common  to  many  organizations. 
Measures  of  Project  Perfonnance  included 

•  EVMS  cost  performance  index  (CPI) 

•  EVMS  schedule  performance  index  (SPI) 

•  Percent  of  key  performance  parameters  (KPP)  satisfied 

•  Percent  of  requirements  satisfied 

•  Percent  of  available  award  fees  received 

Respondents  were  asked  to  provide  data  for  any  or  all  of  these  measures. 

2.5  STEP  5:  DEVELOP  THE  SURVEY  INSTRUMENT 

Defining  characteristics  of  the  survey  instrument  included 

•  integrity  (Respondents  were  assured  that  the  results  of  the  survey  would  be  used  only  for  the 
stated  purposes  of  the  SEEC.) 

•  confidentiality  (Respondents  were  guaranteed  that  their  responses  were  kept  in  confidence.) 

•  self-administration  (Respondents  were  able  to  execute  the  survey  instrument  independently, 
without  intervention  or  involvement  of  the  SEEC.) 

•  self-checking  (The  questionnaire  included  cross-checks  to  ascertain  consistency  and  validity 
of  responses.) 

The  survey  instrument  consisted  of  142  questions  in  three  sections. 

The  first  section  gathered  information  used  to  characterize  the  responding  project.  Fifty-five  ques¬ 
tions  characterized  the  projects  in  terms  of 


project  size 

(resources,  value,  etc.) 

end-user  category 

project  status  (current  life  cycle 
phase,  percent  complete,  etc.) 

organizational  process  focus 


project  stability 

application  domain 
project  team  prior  experience 

process  improvement  activities 


customer  category 

technology  domain 
organizational  experience 
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The  second  section  collected  information  regarding  the  performance  of  Systems  Engineering  ac¬ 
tivities  and  the  production  of  Systems  Engineering  artifacts.  Sixty-five  questions  measured  Sys¬ 
tems  Engineering  performance  in  the  following  areas: 


process  definition 
requirements  development 
interfaces 

test  and  verification 


•  project  planning 

•  requirements  management 

•  product  structure 

•  validation 


•  risk  management 

•  trade  studies 

•  product  integration 

•  configuration  management 


Most  questions  in  this  section  were  structured  in  the  form  of  an  assertion  regarding  the  project 
being  surveyed: 

This  project  has  a  <work product>  with  <defmed  characteristics> 

where:  <work product>  references  one  of  the  CMMI  work  products  identified  for  inclu¬ 
sion  in  the  survey 

<defined  characteristics>  address  the  contents  of  the  work  product 

The  respondent  was  then  asked  to  identify  his  level  of  agreement  with  this  assertion,  choosing 
from  choices  of  strongly  disagree,  disagree,  agree,  or  strongly  agree.  Four  response  options  were 
chosen  to  force  respondents  to  “take  a  position,”  rather  than  choose  a  neutral  response. 

The  third  section  collected  information  on  Project  Performance  using  22  questions. 


•  earned  value  •  award  fee  •  milestone  satisfaction 

•  technical  requirements  •  problem  reports 

satisfaction 


Many  of  these  questions  asked  for  quantitative  data  from  the  project. 

2.6  STEP  6:  DESIGN  THE  SURVEY  EXECUTION  PROCESS 

A  primary  objective  of  the  survey  execution  process  was  to  maximize  the  number  of  qualified 
responses.  This  was  accomplished  in  two  steps: 

•  optimize  sampling 

•  maximize  response  rate 

Sample  size  was  maximized  using  the  resources  of  NDIA  to  reach  a  broad  constituency,  as  dis¬ 
cussed  in  Section  2.2.  The  intent  was  to  reach  a  significant  percentage  of  the  projects  being  exe¬ 
cuted  by  these  organizations. 
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Three  factors  were  used  to  maximize  response  rate. 


First,  we  made  responding  to  the  survey  simple  and  convenient  by  using  the  Web.  To  participate, 
a  respondent  merely  had  to  obtain  an  online  account  from  the  survey  server,  log  in,  and  complete 
the  survey. 

Second,  we  established  data-handling  policies  to  mitigate  respondents’  concerns  about  confidenti¬ 
ality.  Some  organizations  were  expected  to  be  reluctant  to  respond  due  to  the  survey’s  request  for 
competition-sensitive  information  identifying  Project  Performance.  The  following  principles  of 
confidentiality,  trustworthiness,  and  security  were  deployed  throughout  the  survey  and  clearly 
communicated  to  all  participants: 

•  Data  would  be  used  only  for  the  stated  purposes  of  the  survey. 

•  Data  would  be  collected  and  handled  by  a  trusted  organization. 

•  All  responses  would  be  collected  anonymously.  The  survey  would  not  solicit  information  to 
identify  people,  projects,  or  organizations.  Furthermore,  all  respondents  would  be  solicited 
by  proxy,  with  no  contact  between  the  respondents  and  the  surveyor 

•  Data  would  be  collected  and  stored  securely  in  an  encrypted  format. 

•  Data  presented  in  reports  would  include  only  aggregate  data  and  would  not  include  any  in¬ 
formation  traceable  to  any  person,  project,  or  organization. 

The  intent  was  to  convince  respondents  that  they  could  respond  honestly  to  the  survey  questions, 
without  fear  of  exposing  critical  information. 

Respondents  were  identified  and  solicited  by  proxies  within  each  organization.  The  use  of  prox¬ 
ies  ensured  that  respondents  were  contacted  only  by  members  of  their  own  organization.  Our  ex¬ 
pectation  was  that  this  would  improve  the  response  rate.  Flowever,  at  the  same  time,  the  use  of 
proxies  precluded  the  surveyors  from  soliciting  respondents,  from  expediting  responses,  and  from 
knowing  who  had  responded.  Instead,  the  surveyors  had  to  rely  upon  the  proxies  for  these  efforts. 
This  forced  the  SEEC  to  develop  a  communication  and  survey  execution  process  as  shown  in 
Figure  4. 

Third,  the  organizations  and  respondents  needed  an  incentive  to  respond.  We  were  asking  them  to 
spend  time  and  effort  responding  to  the  survey.  In  spite  of  all  of  our  arrangements  for  security  and 
anonymity,  we  were  asking  them  to  take  a  risk,  albeit  a  small  one,  in  exposing  competition- 
sensitive  information.  Some  reward  for  participation  was  needed;  altruism  to  advance  understand¬ 
ing  of  the  field  of  Systems  Engineering  would  not  be  sufficient.  But  offering  some  type  of  reward 
to  anonymous  participants  was  a  difficult  task. 

The  solution  was  to  offer  information  and  knowledge  as  a  reward  for  survey  participation.  If  suc¬ 
cessful,  the  survey  would  provide  a  benchmark  for  SE  performance  among  a  broad  range  of  gov¬ 
ernment  suppliers.  Organizations  could  compare  themselves  against  this  benchmark  and  develop 
process  improvement  plans  to  obtain  a  competitive  advantage.  Access  to  this  benchmark  informa¬ 
tion  would  be  offered  as  a  reward  for  participating  in  the  survey.  Survey  participants  would  re¬ 
ceive  access  to  the  aggregated  survey  data  immediately  upon  its  release.  The  data  would  be  with¬ 
held  from  the  broader  public  for  one  year.  The  Web-based  nature  of  the  survey  execution  also 
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made  it  possible  to  provide  this  information  to  the  respondents  even  while  maintaining  their  ano¬ 
nymity.  To  participate  in  the  survey,  the  respondents  applied  to  the  SEI  Web  server  for  an  account 
name  and  password;  a  password  that  they  could  then  change.  With  this  account  name  and  pass¬ 
word,  they  could  log  in  to  the  Web  server  and  complete  the  survey  in  complete  anonymity.  After 
completion  of  the  survey  analysis,  the  report  could  then  be  posted  on  the  Web  server  accessible 
only  to  those  with  account  names  and  passwords  from  survey  completion.  In  this  manner,  respon¬ 
dents  could  acquire  access  to  the  report  without  loss  of  anonymity. 
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SEEC  Activities 


Figure  4:  Survey  Execution  Method 


14  |  CMU/SEI-2007-SR-014 


NDIA 


3  Survey  Instrument  Testing 


3.1  STEP  7:  PILOT  THE  SURVEY  EXECUTION 

Members  of  the  SEEC  presented  the  survey  to  program  staff  within  their  organizations  for  testing. 
The  focus  of  this  testing  was  fourfold: 

1.  Verify  the  clarity  and  understandability  of  the  survey  questions. 

2.  Assess  the  time  needed  to  complete  the  questionnaire. 

3.  Verify  the  operability  and  reliability  of  the  Web-based  collection  process. 

4.  Verify  the  clarity  and  understandability  of  the  survey  instructions. 

The  responses  represented  the  characteristics  of  real  programs,  verifying  that  the  Web-based  col¬ 
lection  process  worked  effectively.  We  also  held  follow-up  discussions  with  the  beta  respondents 
to  verify  that  they  understood  the  questions  and  responded  appropriately.  In  this  manner,  we  veri¬ 
fied  that  both  the  survey  instructions  and  the  survey  questions  were  clear  and  understandable.  Fi¬ 
nally,  we  asked  the  beta  respondents  to  keep  track  of  the  amount  of  time  required  to  complete  the 
survey  (we  wanted  the  time  kept  below  an  hour — anything  more  would  likely  reduce  the  response 
rate  significantly). 

3.2  STEP  8:  INCORPORATE  FINDINGS  FROM  THE  PILOT 

Results  of  the  pilot  testing  showed  the  following: 

1 .  The  pilot  respondents  found  the  survey  questions  to  be  both  clear  and  understandable.  Dis¬ 
cussions  with  the  respondents  did  not  uncover  any  misinterpretations. 

2.  The  time  needed  to  complete  the  questionnaire  varied  considerably  among  the  respondents. 
Some  completed  the  questionnaire  in  as  little  as  30  minutes.  Others  required  in  excess  of 
three  hours. 

3.  The  Web-based  collection  process  experienced  a  number  of  difficulties  during  the  pilot. 

4.  The  survey  instructions  were  found  to  be  clear  and  understandable. 

The  SEEC  addressed  the  response  time  and  the  Web-based  collection  issues. 

Through  discussions  with  the  pilot  respondents,  the  SEEC  investigated  the  sources  of  completion 
time  variability.  Many  of  the  questions  within  the  questionnaire  require  multiple-choice  re¬ 
sponses.  In  most  cases,  these  were  found  to  be  quickly  and  easily  answerable.  The  wide  variation 
in  completion  times  was  found  to  arise  from  questions  requiring  a  numeric  response.  These  ques¬ 
tions  were  found  predominantly  in  the  first  section  (Project  Characterization)  and  third  section 
(Project  Performance)  of  the  questionnaire.  The  information  provided  by  these  types  of  questions 
was  thought  to  be  very  valuable.  However,  when  considering  the  difficulty  in  responding  to  these 
questions,  the  SEEC  recognized  that  changes  were  needed. 
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•  Some  respondents  felt  that  asking  for  a  numeric  response  inferred  a  heightened  request  for 
precision.  Some  of  the  excessive  response  time  was  spent  in  researching  the  “exact”  numeric 
value  to  be  provided.  In  reality,  the  purposes  of  this  survey  could  be  served  with  responses  of 
low  to  moderate  precision.  To  address  this  finding,  some  of  the  questions  were  reformatted  to 
“quantize”  the  responses.  Instead  of  asking  for  a  numeric  response,  the  respondent  was  asked 
to  choose  among  pre-defined  ranges  of  numeric  responses.  This  clearly  indicated  our  intent 
regarding  precision,  and  significantly  reduced  the  time  required  to  complete  these  questions. 

•  Some  of  the  questions  soliciting  numeric  responses  were  simply  too  difficult  to  answer  for 
some  of  the  respondents.  The  information  being  requested  was  not  readily  available,  and  re¬ 
quired  too  much  research  to  find.  While  we  felt  that  the  requested  information  would  add  val¬ 
ue  to  the  survey,  when  balanced  with  the  anticipated  reduction  in  response  rate  resulting  from 
the  difficulty  in  responding,  we  chose  to  eliminate  many  of  these  questions. 

While  the  total  number  of  questions  remained  essentially  unchanged,  responses  to  the  questions 
were  substantially  simplified.  Additional  pilot  testing  showed  that  completion  time  for  the  revised 
questionnaire  ranged  from  30  minutes  to  1  hour.  This  was  accepted  by  the  SEEC. 

During  the  pilot  testing,  a  number  of  issues  were  found  with  the  Web-based  collection  process.  In 
some  cases,  the  respondent’s  network  security  settings  prevented  them  from  gaining  access  to  the 
survey  portal  and  the  survey  Web  sites.  In  some  cases,  Web  browser  incompatibilities  compro¬ 
mised  the  respondent’s  ability  to  participate.  In  some  cases,  Web  server  errors  prevented  online 
respondents  from  resuming  an  interrupted  response  session  as  planned.  All  of  these  issues  were 
researched,  resolved,  and  tested  during  the  pilot  phase. 

The  resulting  survey  instrument  can  be  seen  in  APPENDIX  B. 
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4  Survey  Execution 


4.1  SOLICITING  RESPONDENTS 

As  noted  above,  the  surveyors  contacted  the  respondents  only  through  proxies  to  protect  their 
anonymity.  After  the  organizations  to  be  surveyed  were  identified  (see  Section  2.2),  the  SEEC 
searched  the  NDIA  SED  “active”  roster  to  identify  contacts  within  each  organization,  with  the 
intent  of  finding  someone  to  act  as  both  an  advocate  for  the  survey,  as  well  as  a  proxy  to  identify, 
contact,  and  interface  with  respondents  within  the  organization.  The  SEEC  also  collaborated  with 
other  organizations  (such  as  ALA.,  IEEE)  to  identify  these  advocates.  Criteria  for  selection  of  these 
designated  “focals”  were  as  follows: 


Organizational  criteria 

Focal  criteria 

•  participant  in  the  supply  chain  of  the  Department  of 

•  holds  a  senior  management  position  within  the  or- 

Defense  (DoD) 

ganization. 

•  delivering  products  to  the  DoD 

•  has  access  to  project  managers  engaged  with  de- 

•  major  operations  with  the  United  States 

fense  contracts  throughout  the  entire  organization. 

•  current  member  of  NDIA 

•  has  sufficient  influence  within  the  organization  to 
encourage  project  managers  to  participate  in  this 

survey. 

•  recognizes  the  importance  of  Systems  Engineering 

and  supports  this  survey  activity. 

The  survey  was  first  introduced  to  the  NDIA  SED  at  the  August  2005  division  meeting.  Subse¬ 
quently,  the  SEEC  proceeded  to  contact  the  candidate  focals  at  the  50  organizations  identified  in 
Section  2.2.  Contacts  were  made  via  face-to-face,  telephone,  and  email  to  explain  the  purpose  and 
the  principles  of  this  survey,  and  solicit  their  support. 

Of  the  contacts  identified  from  the  NDIA  SED  Active  Members  list,  approximately  8%  were  un¬ 
able  to  be  contacted;  some  due  to  inaccurate  contact  infonnation,  some  due  to  mergers  and  acqui¬ 
sitions. 

Another  6%  declined  to  participate  in  the  survey.  One  focal  cited  a  continuing  concern  about  data 
confidentiality.  Another  noted  that  he  felt  no  incentive  to  participate.  A  third  cited  a  general  mis¬ 
trust  of  surveys. 

The  SEEC  contacted  the  remainder  of  the  respondents  repeatedly.  Most  agreed  to  participate.  A 
few  were  non-committal.  Ultimately,  a  data  package  was  sent  to  the  focals  (see  APPENDIX  C). 
This  data  package  consisted  of 

•  a  letter  of  invitation  from  NDIA 

•  the  survey  non-disclosure/privacy  policy 

•  instructions  on  selecting  projects  to  participate  in  the  survey 

•  instructions  to  the  respondent 
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definition  of  terms 


The  instructions  to  the  focals  were  to 

•  identify  appropriate  respondents  within  their  organization  for  the  survey 

•  report  the  number  of  identified  respondents  to  the  SEI 

•  contact  the  identified  respondents,  and  solicit  their  participation  in  the  survey 

•  periodically  expedite  respondents 

•  periodically  report  progress  (i.e.,  the  number  of  responses  submitted)  to  the  SEI 

4.2  RESPONDING  TO  THE  SURVEY 

The  SEI  prepared  to  collect  anonymous  and  confidential  questionnaire  responses  from  the  re¬ 
spondents  via  the  survey  Web  site.  The  Web  site  was  developed  in  a  manner  that  minimized  the 
burden  on  the  respondents.  Upon  logging  on,  the  respondent  received  a  unique  and  randomly  gen¬ 
erated  URL  at  which  he  could  access  a  copy  of  the  questionnaire.  The  respondent  could  access  the 
online  questionnaire  at  the  uniquely  assigned  URL  received  from  the  survey  portal.  Access  to  this 
secure  site  required  both  knowledge  of  the  URL  and  a  user-defined  password.  In  this  manner,  on¬ 
ly  the  respondent  could  access  their  assigned  Web  site.  The  respondent  could  then  complete  the 
questionnaire  online,  saving  his  or  her  results  incrementally.  At  any  time,  the  respondent  could 
exit  the  Web  site  without  losing  the  data  saved  to  date.  In  this  manner,  the  respondent  could  com¬ 
plete  the  questionnaire  over  multiple  sessions.  On  completion  of  the  questionnaire,  the  respondent 
notified  the  survey  server  by  clicking  on  the  ‘Submit’  button. 

The  SEI  began  to  receive  responses  shortly  after  the  focals  were  contacted. 

As  with  any  survey,  response  expediting  was  necessary.  The  solicitation  of  respondents  via  prox¬ 
ies  complicated  this  process.  With  the  actual  respondents  unknown  to  the  SEEC,  the  SEEC  could 
only  ask  the  focals  to  expedite  the  respondents.  About  two  weeks  after  the  start  of  data  collection, 
the  SEEC  emailed  the  focals,  asking  them  to 

•  check  with  project  leaders  to  see  which  have  responded 

•  expedite  non-responders 

•  notify  SEI  of  the  number  of  projects  which  have  responded  to  date 

This  expediting  effort  was  repeated  approximately  two  weeks  later  and  again  two  weeks  after  that. 

Obtaining  response  data  from  the  focals  was  not  highly  effective.  Our  intent  was  to  keep  track  of 
response  rates  by  identifying  the  number  of  respondents  solicited  by  each  focal,  and  the  number 
of  responses  reported  complete  by  each  focal.  Even  after  numerous  contacts  of  the  focals,  we 
were  unable  to  collect  sufficient  data  to  support  this  goal. 

The  survey  Web  server  accepted  responses  from  August  10,  2006  until  November  30,  2006.  Dur¬ 
ing  this  period  64  surveys  were  collected.  Upon  review  of  the  responses,  it  was  clear  that  several 
were  initiated  but  not  completed.  These  were  discarded,  resulting  in  46  valid  survey  responses. 
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5  Analysis 


The  primary  survey  hypothesis  has  been  stated  as  follows: 

The  effective  performance  of  SE  best  practices  on  a  development  program 
yields  quantifiable  improvements  in  the  program  execution  (for  example, 
improved  cost  performance,  schedule  performance,  technical  performance). 

Mathematically,  we  can  state  this  as 


Perf=f  (PC,  PE,  SEC,  AC) 


where:  Project  Challenge 

Project  Environment 
Systems  Engineering  Capability 


PC 

PE 

SEC 

AC 

Perf 


Acquirer  Capability 
Project  Performance 


More  detailed  descriptions  of  each  of  these  factors  are  found  within  this  section. 

Our  goal  is  to  identify  the  impact  of  PC,  PE,  SEC,  and  AC  upon  Perf.  We  can  do  this  by  identify¬ 
ing  the  relationships  among  each  of  these  factors  and  Perf.  The  primary  objective  is  to  identify  the 
statistical  association  between  SEC  and  Perf.  We  will  consider  AC,  PE,  and  PC  as  factors  moder¬ 
ating  these  primary  relationships. 

Each  of  these  measures  is  derived  by  combining  the  responses  for  a  set  of  conceptually  related 
questions.  Because  the  individual  questions  can  be  interpreted  somewhat  differently  by  different 
survey  respondents,  combining  the  responses  into  an  overall  composite  measure  reduces  the  unre¬ 
liability  associated  with  any  single  question  [Guilford  1954].  These  composite  measures  are 
weighted,  summed  indices  of  the  responses  to  each  set  of  questions  from  each  participating  pro¬ 
ject.  For  example,  many  of  the  response  categories  range  ordinally  from  “disagree  strongly”  to 
“agree  strongly.”  The  projects’  answers  are  scored  as  1  through  4  respectively  and  then  summed. 
Since  the  number  of  component  items  varies  for  each  of  the  composite  measure,  the  scores  are 
normalized  to  allow  consistent  interpretation  of  their  meaning.  Much  like  student  grade  point  av¬ 
erages,  the  composite  scores  are  divided  by  the  number  of  questions  answered.  The  composite 
scores  thus  are  constrained  to  range  between  1  through  4. 5  Calculating  the  composite  scores  in 
this  manner  provided  sufficient  variation  to  enable  meaningful  statistical  comparisons. 

The  Project  Challenge  ( PC)  questions  address  a  number  of  diverse  issues  contributing  to  the  diffi¬ 
culty  of  a  project;  issues  such  as  project  size,  project  complexity,  technology  precedents,  and  oth¬ 
ers.  All  of  these  factors  are  combined  into  a  single  PC  measure,  with  the  intent  of  examining  the 

5  Such  a  normalization  procedure  is  appropriate  for  ordinal  data  since  the  component  items  fall  in  the  same  con¬ 
strained  range.  Since  the  fractional  differences  cannot  be  interpreted  additively,  the  composite  scores  then  are 
split  into  two  or  three  groupings  as  appropriate  for  the  data  analysis,  (e.g.,  “Lower,"  “Moderate,”  and  “Higher” 
groupings 
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impact  of  the  project  difficulty  upon  Perf  and  the  relationships  between  SEC  and  Perf  (see  Sec¬ 
tion  5.1.1) 

The  Project  Environment  (PE)  measures  address  factors  other  than  Project  Challenge  and  Sys¬ 
tems  Engineering  Capability  that  could  influence  Project  Performance.  These  factors  include  the 
acquiring  organization,  the  end  user,  the  position  in  the  systems  hierarchy,  the  deployment  envi¬ 
ronment,  the  contract  type,  the  percent  of  effort  dedicated  to  Systems  Engineering,  the  develop¬ 
ment  organization’s  CMMI-related  capabilities  (PEcmmi),  the  development  organization’s  process 
improvement  efforts  (PEimp),  and  the  development  organization’s  prior  experience  (PEEXp).  The 
nature  of  these  PE  elements  is  sufficiently  diverse  that  it  is  pointless  to  attempt  to  combine  them 
into  a  single  PE  measure.  Instead,  the  impact  on  Project  Performance  of  each  of  the  PE  elements 
was  evaluated  individually  (see  Section  5.1.2). 

The  questionnaire  was  designed  to  permit  the  Systems  Engineering  Capability  (SEC)  measure  to 
be  decomposed  into  12  measures  of  SE  Capability  in  specific  process  areas: 


IPT-Based  Capability 
Project  Planning 
Project  Monitoring  and  Control 
Risk  Management 

Requirements  Development  and  Management 
Trade  Studies 
Product  Architecture 

Technical  Solution  (=  SEC//(.,H/;  +  SEC  |/«  //) 

Product  Integration 

Verification 

Validation 

Configuration  Management 


(  SECIPT  )  Section  5. 1.3.1 

(SECPP  )  Section  5. 1.3. 2 

(SECPjWc  )  Section  5. 1.3.3 

(  SECrskm  )  Section  5. 1.3. 4 

(SECPPg  )  Section  5. 1.3.5 

(  SECtrade  )  Section  5. 1.3. 6 

(  SEC^flcw  )  Section  5. 1.3.7 

(SECrs  )  Section  5. 1.3. 8 

( SECp/  )  Section  5. 1.3.9 

(SEC  VER  )  Section  5.1.3.10 

(SEC  Mi  )  Section  5.1.3.11 

( SECo/  )  Section  5.1.3.12 


With  this  decomposition,  it  is  possible  to  look  at  more  specific  relationships  between  these  Sys¬ 
tems  Engineering  Capability  factors  and  Project  Performance.  As  noted  previously,  the  work 
products  identified  in  CMMI  were  used  as  the  basis  for  this  survey.  Thus,  the  partitioning  of  the 
SEC  responses  into  categories  similar  to  CMMI  Process  Areas  is  sensible.  Even  though  the  link¬ 
age  between  this  survey  and  CMMI  is  strong,  be  advised  that  although  the  names  of  the  survey 
categories  resemble  those  of  the  model,  they  are  not  perfectly  aligned.  The  survey  categories  do 
not  contain  all  aspects  of  the  similar  CMMI  Process  Areas.  Furthermore,  in  many  cases,  they  con¬ 
tain  extensions  that  are  not  contained  within  the  model.  As  such,  take  care  not  to  “over-interpret” 
the  relationship  between  the  survey  results  and  CMMI. 

The  Acquirer  Capability  (AC)  measure  addresses  the  impact  of  the  acquirer’s  capability  upon  Pro¬ 
ject  Performance.  Because  the  survey  respondents  are  the  project  suppliers,  and  not  the  project 
acquirers,  any  information  gathered  regarding  the  acquirers  is  second-hand  information;  that  is,  it 
is  an  evaluation  of  the  acquirer  from  the  perspective  of  the  supplier.  Nevertheless,  there  are  a  few 
parameters  that  can  be  measured  to  imply  Acquirer  Capability;  parameters  such  as 

•  acquirer’s  participation  on  Integrated  Project  Teams  (IPTs) 

•  acquirer’s  provision  of  a  Systems  Engineering  Plan  (SEP) 

•  quality  of  system  requirements 

•  completeness  of  system  requirements 
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stability  of  system  requirements 


Although  this  survey  was  not  specifically  designed  to  assess  the  capabilities  of  the  acquirers,  these 
parameters  can  be  combined  to  develop  a  rudimentary  measure  of  Acquirer  Capability  ( AC)  (see 
Section  5.2.4). 

Finally,  Project  Performance  ( Perf)  can  be  measured  and  decomposed  into: 


Cost  Performance 

Schedule  (Duration)  Performance 

Scope  Performance 


(  Perfc  ) 
(  Perfo  ) 
(  Perfs  ) 


The  relationship  between  project  cost,  schedule,  and  scope  is  well  known  to  project  managers,  and 
is  commonly  referred  to  as  the  “iron  triangle,”  reflecting  the  fact  that  project  manager  can  often 
modify  the  value  of  one  of  these  parameters,  but  only  at  the  expense  of  the  other  two.  For  exam¬ 
ple,  a  project  manager’s  election  to  reduce  project  cost  will  have  adverse  impacts  upon  the  project 
schedule  and  the  achieved  scope  of  the  project.  As  such,  looking  for  relationships  between  SEC 
and  the  individual  components  of  Perf  (i.e.,  Perfc,  PerfD,  and  Perfs)  would  not  be  as  useful  as 
looking  for  relationships  between  SEC  and  a  composite  Project  Performance  variable  combining 
all  three  of  these  components  (see  Section  5. 1.5.4). 

5.1  RESPONDENT  PROFILE 

To  profile  the  responding  project,  the  survey  requested  information  about 

•  the  project 

•  the  product  resulting  from  the  project 

•  the  contract  establishing  the  project 

•  the  organization  executing  the  project 

•  the  Systems  Engineering  practices  deployed  on  the  project 

•  Project  Performance  data 

Responses  were  analyzed  to  identify  Project  Challenge,  Project  Environment,  Project  Systems 
Engineering  Capability,  and  Project  Performance. 

Responses  are  presented  as  distribution  graphs  showing  the  frequency  of  each  response,  as  seen  in 
Figure  5  and  Figure  6.  Figure  5  is  an  example  of  the  results  from  a  single  question.  It  has  the  fol¬ 
lowing  characteristics: 

•  The  horizontal  axis  shows  the  survey’s  available  response  choices 

•  The  vertical  bars  represent  the  percentage  of  total  respondents  selecting  each  response  choice 

Figure  6  is  an  example  of  the  results  of  a  composite  score  calculated  from  the  combination  of  re¬ 
sponses  to  multiple  related  questions.  It  has  the  following  characteristics: 

•  The  horizontal  axis  represents  the  range  of  the  composite  score,  usually  scaled  from  1  to  4. 
For  a  graph  depicting  an  SE  Capability,  1  would  represent  a  low  capability  and  4  would  rep- 
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resent  a  high  capability.  For  a  graph  depicting  Project  Challenge,  1  would  represent  low  chal¬ 
lenge  while  4  would  represent  high  challenge. 

•  The  horizontal  range  of  1  to  4  is  divided  into  a  number  of  equally  sized  bins.  The  vertical  bars 
represent  the  percentage  of  respondents  whose  score  falls  within  the  range  of  each  bin. 


Figure  5:  Example  Distribution  Graph  for  an  Individual  Question 


Histogram  of  response 
frequencies 


f 


Outliers 


2 


± 


± 


um  =  3.6 
d  Quartile  =  2.8 
Median  =  2.4 
1st  Quartile  =  2.2 
Minimum  =  1.0 
N  =  64 


interquartile  range 

V- 


median 


Data  range 


Figure  6:  Example  Distribution  Graph  for  a  Composite  Score 
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The  graphic  on  the  bottom  of  Figure  6  is  known  as  an  outlier  box  plot.  It  visually  shows  the  range 

and  concentration  of  the  full  distribution  of  composite  scores. 

•  The  interquartile  range  is  the  space  between  the  upper  and  lower  quartiles,  which  also  are 
known  as  the  75th  and  25th  quartiles,  respectively.  It  contains  50%  of  all  the  cases. 

•  The  solid  lines  extending  from  the  box  are  called  “whiskers.”  Their  ends  extend  to  the  outer¬ 
most  data  points  that  are  not  outliers. 

•  Outliers  are  defined  as  data  points  that  are  greater  than  +1.5  times  the  size  of  the  interquartile 
range. 

•  Narrower  interquartile  boxes,  shorter  whiskers  and  the  absence  of  outliers  indicate  more  con¬ 
sistency  among  the  scores.  The  opposite  indicates  more  variability. 

5.1.1  Project  Challenge  (PC) 

The  survey  estimated  the  degree  of  challenge  posed  by  the  project  through  a  combination  of  fac¬ 
tors  including: 


•  included  life-cycle  phases 

•  sources  of  technical  challenge 

•  inter-organizational  complexity 

•  contract  duration 

•  contract  stability  (number 
of  change  orders) 

•  change  in  contract  duration 


•  life-cycle  phases  currently  in  execution 

•  total  project  effort 

•  contract  value 

•  requirements’  completeness  and  stability 

•  percent  change  of  contract  value 

•  dollar  change  of  contract  value 


This  information  was  collected  through  responses  to  questions  ProjOl,  Proj02,  Proj08,  ContOl 
through  Cont07,  ContlO  through  Contl2,  and  Contl4. 


ID 

Question 

Response  range 

ProjOl 

What  phases  of  the  integrated  product  life  cycle  are  or  will  be  in¬ 
cluded  in  this  project? 

Scored  by  the  number  of  life- 
cycle  phases  included 

•  concept  refinement 

•  technology  dev’t  &  demo 

•  development 

•  mf’g  /  production 

•  verification  /  validation 

•  training 

•  deployment 

•  operation 

•  support 

•  disposal 
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ID 

Question 

Response  range 

Proj02 

What  phase  or  phases  of  the  integrated  product  life  cycle  is  this 
project  presently  executing? 

Scored  by  the  number  of  life- 
cycle  phases  in  execution 

•  concept  refinement 

•  technology  dev’t  &  demo 

•  development 

•  mfg  /  production 

•  verification  /  validation 

•  training 

•  deployment 

•  operation 

•  support 

•  disposal 

Proj08 

The  project  is  technically  challenging  because... 

Scored  by  the  number  of 
challenges  noted 

•  no  precedent 

•  quality  attribute  constraints 

•  large  development  effort 

•  immature  technology 

•  extensive  interoperability 

•  insufficient  resources 

•  insufficient  skills 

ContOI 

What  is  the  current  total  contract  value  of  this  project? 

•  <$  10  M 

•  <$  100M 

•  <$  1  B 

•  <$  10  B 

•  >$  10  B 

Cont02 

What  is  the  current  total  planned  duration  of  this  project? 

•  <12  months 

•  12-24  months 

•  24-48  months 

•  48-96  months 

•  96-192  months 

•  >192  months 

Cont03 

What  was  the  initial  contract  value  of  this  project? 

•  <$  10  M 

•  <$  100M 

•  <$  1  B 

•  <$  10  B 

•  >$ 10  B 

Cont04 

What  was  the  initial  total  planned  duration  of  this  project? 

•  <12  months 

•  12-24  months 

•  24-48  months 

•  48-96  months 

•  96-192  months 

•  >192  months 

Cont05 

How  many  contract  change  orders  have  been  received? 

•  <=  1 

•  <=10 

•  <=  100 

•  <=1000 

•  >  1000 

Cont06 

Approximately  how  many  person-years  of  effort  are  allocated  to  be 
spent  on  this  project  within  your  organization? 

•  <  10 

•  <  50 

•  <200 

•  <  2000 

•  >  2000 
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ID 

Question 

Response  range 

Cont07 

What  program  acquisition  category  (ACAT  level)  is  your  program 
classified  at? 

•  Don't  Know 

•  ACAT  IAC 

.  ACAT  1AM 

.  ACAT  1C 

.  ACAT ID 

.  ACAT  II 

.  ACAT  III 

•  Other 

ContIO 

Flow  many  stakeholders  (including  internal  and  external)  are  in¬ 
volved  in  this  project? 

Numeric  entries  for  each  of 
the  following  stakeholder 
categories 

•  acquirers 

•  SI  contractors 

•  maintenance  contractors 

•  dev’t  co-contractors 

•  dev’t  sub-contractors 

•  oversight  contractors 

•  users 

•  others 

Entries  were  quantized  as  1 , 

2,  3,  or  >3 

Conti  1 

What  percentage  of  the  customer  technical  requirements  were 
marked  “To  Be  Determined”  at  time  of  contract  award? 

•  <1% 

•  1-5% 

•  5-20% 

•  >20% 

Cont12 

What  percentage  of  the  customer's  technical  requirements  are  cur¬ 
rently  marked  “To  Be  Determined”? 

•  <1% 

•  1-5% 

•  5-20% 

•  >20% 

Cont14a 

Approximately  what  percentage  of  non-recurring  engineering  (NRE) 
does  systems  engineering  represent? 

0  to  1 00%  quantized  as 

•  0-5% 

.  5-5  10% 

•  10-15% 

•  15-25% 

•  >25% 

Cont14b 

Is  the  NRE  percentage  estimated,  or  is  it  a  measured  value? 

•  Estimated 

•  Measured 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  a  measure  of  Project  Challenge — PC.  Distribution  of  PC  is  seen  in  Figure  7. 
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Maximum  =  2 

.8 

3rd  Quartile  = 

2.1 

Median  =  1 .9 

1st  Quartile  = 

1.7 

Minimum  =  1 

1 

N  =  64 

Figure  7:  PC  Composite  Measure 

This  analysis  reveals  that  the  64  projects  responding  to  this  survey  were  not  unusually  complex. 
On  a  complexity  scale  of  1  to  4  half  of  the  projects  ranged  from  1.1  to  1.9,  and  the  other  half 
ranged  from  1.9  to  2.8. 


Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.l. 

5.1.2  Project  Environment  (PE) 

Factors  other  than  Project  Challenge  may  also  influence  Project  Performance.  Factors  considered 
in  the  survey  included: 

•  the  customer  type 

•  the  acquiring  organization 

•  the  end  user 

•  the  position  in  systems  hierarchy 

•  the  deployment  environment 

•  the  contract  type 

•  the  percent  of  effort  subcontracted 

•  the  development  organizations  CMMI-related  capabilities 

•  the  development  organization’s  process  improvement  efforts 

•  the  development  organization’s  prior  experience 

This  information  was  collected  through  responses  to  questions  ProdOl  through  Prod05,  Proj09, 
ContOS,  Conti 5,  OrgOl  through  Org05,  and  Org07.  Response  distributions  are  shown  below. 
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As  seen  in  Figure  8,  the  responding  project’s  customers  were  primarily  the  government  or  the 
prime  contractors;  thus  the  respondent’s  projects  were  being  executed  either  by  the  prime  contrac¬ 
tor  or  a  second-tier  subcontractor,  with  nearly  70%  of  the  responses  provided  by  the  prime  con¬ 
tractors. 


70- 

60- 

50- 


I  40-| 

V 

“■  30-1 


20- 

10 

0 


US  Government  Prime  Contractor 

S3Q1  PE01  ProdOl 


Subcontractor 


Figure  8:  ProdOl  -  Which  Selection  Best  Characterizes  Your  Customer? 

The  distribution  of  the  acquiring  organizations  is  seen  in  Figure  9.  A  large  number  of  the  acquir¬ 
ers  are  contained  in  the  “Other”  category.  A  review  of  the  responses  indicates  that  these  acquirers 
are  primarily  foreign  governments.  The  projects  in  the  “Commercial”  category  are  those  that  list 
the  customer  as  a  “Prime  Contractor”  or  “Subcontractor”  in  Figure  8.  The  questionnaire  collected 
no  information  to  further  identify  those  projects  in  the  “Other  Government”  category. 

Figure  10  shows  the  distribution  of  the  project’s  end  users;  a  distribution  substantially  similar  to 
the  acquirer  distribution  of  Figure  9.  Note  that  the  bars  on  the  graph  sum  to  greater  than  100%. 
This  is  due  to  the  fact  that  some  projects  have  more  than  one  set  of  end-users 
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Figure  9:  Prod02  -  Who  Is  Acquiring  This  Product? 
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Figure  1 0:  Prd03  -  Who  Is  Primary  End  User  (or  Users)  of  This  Product? 

Figure  1 1  shows  the  position  of  the  delivered  product  in  a  systems  hierarchy  divided  into  catego¬ 
ries: 
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•  system  of  systems 

•  system 

•  subsystem 

•  component 

•  process 

•  material 

•  other 

While  the  definitions  of  these  terms  are  somewhat  nebulous,  our  intent  was  to  try  to  characterize 
the  projects  in  a  manner  that  would  enable  us  to  identify  the  importance  of  Systems  Engineering 
for  each  category.  Unfortunately,  the  number  of  responses  received  was  not  sufficient  to  enable  a 
meaningful  analysis. 

As  seen  in  Figure  11,  most  of  the  respondents  classify  their  products  as  a  system.  A  smaller  but 
not  insignificant  number  consider  their  products  to  be  systems-of-systems. 


Figure  1 1:  Prod04  -  In  the  Context  of  the  Ultimate  Product  Delivered  to  the  End  User,  Where  Does  This 
Project  Fit  in  the  Following  Hierarchy? 

The  distribution  of  the  deployment  environment  is  seen  in  Figure  12.  The  projects  contained  with¬ 
in  the  “Other”  category  were  primarily  deployed  in  combinations  of  the  other  environments. 
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Figure  12:  Prod05  -Where  Will  the  System  Resulting  From  This  Project  be  Used? 

Project  execution  will  fall  within  a  continuum  ranging  from 

•  The  project  is  executed  entirely  by  the  prime  contractor;  to 

•  The  project  is  entirely  subcontracted  to  other  suppliers. 

Figure  13  shows  the  distribution  of  the  responses.  These  responses  were  then  binned  in  categories 
of  0  to  5%,  5  to  25%,  25  to  50%,  and  greater  than  50%,  as  seen  in  Figure  14. 


Subcontracting 


Subcontracted  %  of  Contract  Value 


Figure  13:  Distribution  of  Subcontracted  Effort 
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Figure  14:  Cont08  -  What  Percentage  of  the  Total  Contract  Value  is  Subcontracted  to  Your  Suppliers? 

Various  contract  types  may  be  executed  for  a  project;  in  fact,  a  project  may  actually  include  mul¬ 
tiple  contracts  of  different  types.  Respondents  were  asked  which  of  the  following  types  of  con- 


tracts  applied  to  their  projects: 

FFP: 

Firm  fixed  price  — FAR  16.202 

FP+EPA: 

Fixed  price  with  economic  price  adjustment  —  FAR  16.203 

FP+PPR: 

Fixed  price  with  prospective  price  redetermination  -  FAR  16.205 

FP+RPF: 

Fixed  ceiling  with  retroactive  price  redetermination  —  FAR  16.206 

FFP,  LOE: 

Firm  fixed  price,  level  of  effort  -  FAR  16.207 

CR: 

Cost  reimbursement  — FAR  16.302 

CS: 

Cost  sharing  — FAR  16.303 

CPIF: 

Cost  plus  incentive  fee  —  FAR  16.304 

CPFF: 

Cost  plus  fixed  fee  -  FAR  16.306 

FPIF: 

Fixed  price  incentive  —  FAR  16.403 

FPAF: 

Fixed  price  with  award  fees  —  FAR  16.404 

CPAF: 

Cost  plus  award  fee  -  FAR  16.405 

Other 
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Figure  1 5:  Conti 5  -  What  Type  of  Contract(s)  was  Awarded  for  This  Project? 

As  seen  in  Figure  15,  contract  types  varied  across  the  project  sample  with 
19%  being  FFP:  Firm  fixed  price  —  FAR  16.202 

2%  being  FP+EPA:  Fixed  price  with  economic  price  adjustment  —  FAR  16.203 
5%  being  FFP,  LOE:  Firm  fixed  price,  level  of  effort  —  FAR  16.207 
8%  being  CR:  Cost  reimbursement  —  FAR  16.302 

Cost  sharing  -  FAR  16.303 
Cost  plus  incentive  fee  —  FAR  16.304 
Cost  plus  fixed  fee  —  FAR  16.306 
Fixed  price  incentive  —  FAR  16.403 
14%  being  FPAF:  Fixed  price  with  award  fees  —  FAR  16.404 
34%  1  being  CPAF:  Cost  plus  award  fee  —  FAR  16.405 
16%  being  Other 

This  analysis  reveals  that  most  of  the  contracts  were  some  form  of  cost-reimbursable  contract. 


3%  being  CS: 
27%  being  CPIF: 
23%  being  CPFF: 
9%  being  FPIF : 


5.1. 2.1  CMMI-Related  Project  Environmental  Factors  (PEcmmi) 


The  responding  project’s  capabilities  related  to  CMMI  varied  across  a  wide  range.  Overall, 
CMMI-related  capability  was  reported  as  moderate  Capability  in  regards  to  CMMI  was  identified 
through  questions  Org02,  Org04,  Org05,  and  Org06. 


ID 

Question 

Response  range 

Org02 

To  what  extent  do  the  tailored  processes  followed  by  this  project 
comply  with  the  organization’s  standard  processes? 

•  highly  compliant 

•  largely  compliant; 

•  moderately  compliant 

•  not  compliant 
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ID 

Question 

Response  range 

Org04 

At  what,  if  any,  CMM  or  CMMI  Maturity  Level  has  this  project's 
parent  organization  most  recently  been  appraised? 

•  not  appraised 

•  ML1 

•  ML2 

•  ML3 

•  ML4 

•  ML5 

Org05 

When  was  the  organization's  most  recent  appraisal? 

Entered  dates  quantized  as: 

•  <6  mo 

•  <1  yr 

•  <  2yr 

•  >2yr 

Org07 

Flas  this  project  been  objectively  verified  to  be  implementing 
processes  consistent  with  a  given  CMM/CMMI  maturity  level? 

•  Not  Appraised 

•  ML1 

•  ML2 

•  ML3 

•  ML4 

•  ML5 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  PEcmmi ■  Distribution  of  PECmmi  is  seen  in  Figure  16. 


Maximum  =  3.5 
3rd  Quartile  =  2.8 
Median  =  2.5 
1st  Quartile  =  1.7 
Minimum  =  1 .0 
N  =  64 


Figure  1 6:  CMMI-Related  Capability  (PEcmmi)  Composite  Measure 

Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.2.1. 

Contextual  information  not  included  in  the  calculation  of  SECcmmi  was  collected  by  question 
Org06. 

Org06  What  model  was  used? 

( 1  =CMMI-SE/S W/IPPD/S S,  2=CMMI-SE/SW/IPPD,  3=CMMI-SE/SW,  4=CMMI-SW) 
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5.1. 2.2  Prior  Experience  Environmental  Factors  ( PEexp ) 


The  responding  project  indicated  a  moderate  to  high  level  of  prior  experience  on  similar  projects. 
The  prior  experience  of  the  project  team  and  the  organization  was  identified  through  questions 
Proj09  and  OrgOla. 


ID 

Question 

Response  range 

Proj09 

This  project  team  has  successfully  completed  projects  similar  to  this 
in  the  past. 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

OrgOla 

This  organization  has  successfully  completed  projects  similar  to  this 
one  in  the  past. 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  PEEXp.  Distribution  of  PEEXp  is  seen  in  Figure  17. 


Maximum  = 

4.0 

3rd  Quartile 

=  3.5 

Median  =  3 

0 

1st  Quartile 

=  2.5 

Minimum  = 

1.0 

N  =  64 

Figure  17:  Prior  Experience  (PEexp)  composite  measure 

This  analysis  reveals  that  the  respondent  projects  had  relatively  high  levels  of  prior  experience. 
On  an  experience  scale  ranging  from  1  to  4,  half  of  the  projects  ranged  from  1  to  3,  and  the  other 
half  ranged  from  3  to  4. 

Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.2.2. 

5.1. 2. 3  Process  Improvement  Environmental  Factors  ( PEiMp ) 

The  responding  projects  indicated  moderate  Process  Improvement  capability.  The  Process  Im¬ 
provement  capability  of  the  project  team  and  the  organization  was  identified  through  questions 
OrgOlb  and  Org03. 
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ID 

Question 

Response  range 

OrgOlb 

Process  improvement  efforts  in  this  organization  have  been  directly 
related  to  systems  engineering. 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Org03 

What  process  improvement  activities  have  been  undertaken  on  this 
project? 

Scored  by  the  number  of 
process  improvement  meth¬ 
ods  utilized 

•  ISO  9000 

•  Lean 

•  Six  Sigma 

•  SE-CMM 

•  SW-CMM 

•  SECAM 

•  EIA-731 

•  CMMI 

•  none 

•  other 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  PEjmp .  Distribution  of  PEjmp  is  seen  in  Figure  18. 


Maximum  =  3.5 
3rd  Quartile  =  2.8 
Median  =  2.3 
1st  Quartile  =  2.1 
Minimum  =  1.0 
N  =  62 


Figure  1 8:  Process  Improvement  (PEimp)  Composite  Measure 

Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.2.3. 

5.1.3  Systems  Engineering  Capability  (SEC) 

The  survey  collects  data  to  assess  projects’  capabilities  in  each  of  the  categories  defined  in  Sec¬ 
tion  5.  The  responses  to  the  survey  questions  within  each  category  are  analyzed  to  provide  a 
measure  of  project  capability  within  that  category.  This  analysis  is  presented  in  the  form  of  anno¬ 
tated  distributions  as  shown  in  Figure  6. 
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5.1. 3.1  Integrated  Project  Team  Capability  (SECipt) 


The  use  of  Integrated  Project  Teams  by  the  reporting  projects  was  moderate  to  high.  The  use  of 
Integrated  Project  Teams  was  identified  through  questions  Proj03,  Proj04,  Proj06,  Proj07a, 
Proj07b 


ID 

Question 

Response  range 

Proj03 

This  project  uses  integrated  product  teams  (IPTs) 

•  Yes 

•  No 

Proj04 

This  project  makes  effective  use  of  integrated  product  teams  (IPTs) 

•  highly  compliant 

•  largely  compliant; 

•  moderately  compliant 

•  not  compliant 

Proj06 

My  suppliers  actively  participate  in  IPTs 

•  highly  compliant 

•  largely  compliant; 

•  moderately  compliant 

•  not  compliant 

Proj07a 

This  project  has  an  IPT  with  assigned  responsibility  for  systems 
engineering 

•  highly  compliant 

•  largely  compliant; 

•  moderately  compliant 

•  not  compliant 

Proj07b 

This  project  has  Systems  Engineering  representation  on  each  IPT 

•  highly  compliant 

•  largely  compliant; 

•  moderately  compliant 

•  not  compliant 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  SECjpt.  Distribution  of  SEC//>7-is  seen  in  Figure  19. 


Maximum  =  4.0 
3rd  Quartile  =  3.5 
Median  =  3.0 
1st  Quartile  =  2.5 
Minimum  =  1 .0 
N  =  64 


Figure  19:  Integrated  Project  Team  Capability  (SECipt)  Composite  Measure 

Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.3.1. 
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5.1. 3.2  Project  Planning  Capability  (SEC pp) 


Projects  reported  moderate  to  high  application  of  Project  Planning  best  practices.  Application  of 
Project  Planning  best  practices  was  identified  through  questions  PD01  through  PD09 


ID 

Question 

Response  range 

PD01 

This  project  utilizes  a  documented  set  of  systems  engineering  proc¬ 
esses  for  the  planning  and  execution  of  the  project 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD02a 

This  project  has  an  accurate  and  up-to-date  Work  Breakdown 
Structure  (WBS)  that  includes  task  descriptions  and  work  package 
descriptions 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD02b 

This  project  has  an  accurate  and  up-to-date  Work  Breakdown 
Structure  (WBS)  that  is  based  upon  the  product  structure 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD02c 

This  project  has  an  accurate  and  up-to-date  Work  Breakdown 
Structure  (WBS)  that  is  developed  with  the  active  participation  of 
those  who  perform  the  systems  engineering  activities 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD02d 

This  project  has  an  accurate  and  up-to-date  Work  Breakdown 
Structure  (WBS)  that  is  developed  with  the  active  participation  of  all 
relevant  stakeholders,  e.g.,  developers,  maintainers,  testers,  in¬ 
spectors,  etc. 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD03a 

This  project’s  Technical  Approach  (i.e.  a  top-level  strategy  and 
methodology  to  create  the  initial  conceptual  design  for  product  de¬ 
velopment)  is  complete,  accurate  and  up-to-date 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD03b 

This  project’s  Technical  Approach  (i.e.  a  top-level  strategy  and 
methodology  to  create  the  initial  conceptual  design  for  product  de¬ 
velopment)  is  developed  with  the  active  participation  of  those  who 
perform  the  systems  engineering  activities 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD03c 

This  project’s  Technical  Approach  (i.e.  a  top-level  strategy  and 
methodology  to  create  the  initial  conceptual  design  for  product  de¬ 
velopment)  is  developed  with  the  active  participation  of  all  appropri¬ 
ate  functional  stakeholders 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD04a 

This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan 
(IMP),  that  is  an  event-driven  plan  (i.e.,  each  accomplishment  is  tied 
to  a  key  project  event) 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD04b 

This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan 
(IMP),  that  documents  significant  accomplishments  with  pass/fail 
criteria  for  both  business  and  technical  elements  of  the  project 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD04c 

This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan 
(IMP),  that  is  consistent  with  the  WBS 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD05a 

This  project  has  an  integrated  event-based  schedule  that  is  struc¬ 
tured  as  a  networked,  multi-layered  schedule  of  project  tasks  re¬ 
quired  to  complete  the  work  effort 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD05b 

This  project  has  an  integrated  event-based  schedule  that  contains  a 
compilation  of  key  technical  accomplishments  (e.g.,  a  Systems 
Engineering  Master  Schedule) 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 
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ID 

Question 

Response  range 

PD05c 

This  project  has  an  integrated  event-based  schedule  that  refer¬ 
ences  measurable  criteria  (usually  contained  in  the  Integrated  Mas¬ 
ter  Plan)  required  for  successful  completion  of  key  technical  ac¬ 
complishments 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD05d 

This  project  has  an  integrated  event-based  schedule  that  is  consis¬ 
tent  with  the  WBS 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD05e 

This  project  has  an  integrated  event-based  schedule  that  identifies 
the  critical  path  of  the  program  schedule 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD06 

This  project  has  a  plan  or  plans  for  the  performance  of  technical 
reviews  with  defined  entry  and  exit  criteria  throughout  the  life  cycle 
of  the  project 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD07 

This  project  has  a  plan  or  plans  that  include  details  of  the  manage¬ 
ment  of  the  integrated  technical  effort  across  the  project  (e.g.,  a 
Systems  Engineering  Management  Plan  or  a  Systems  Engineering 
Plan) 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD08 

Those  who  perform  systems  engineering  activities  actively  partici¬ 
pate  in  the  development  and  updates  of  the  project  planning 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD09 

Those  who  perform  systems  engineering  activities  actively  partici¬ 
pate  in  tracking/reporting  of  task  progress 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  SECpp.  Distribution  for  the  aggregate  SECPP  is  shown  in  Figure  20. 


Maximum  = 

4.0 

3rd  Quartile 

=  3.4 

Median  =  3 

0 

1st  Quartile 

=  2.6 

Minimum  = 

2.0 

N  =  63 

Figure  20:  Project  Planning  Capability  ( SECpp )  Composite  Measure 
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This  analysis  shows  a  high  level  of  Project  Planning  capability  among  the  responding  projects.  On 
a  Project  Planning  Capability  scale  of  1  to  4,  no  projects  scored  below  2.0.  Half  scored  between 
2.0  and  3.0.  Half  scored  between  3.0  and  4.0. 

Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.3.2. 

5.1. 3. 3  Project  Monitoring  and  Control  Capability  (SEC^ff) 

The  responding  project’s  application  of  Project  Monitoring  and  Control  best  practices  varied  over 
a  wide  range,  with  most  projects  reporting  moderate  to  high  deployment.  Application  of  Project 
Monitoring  and  Control  best  practices  was  identified  through  questions  Conti  3,  Conti  4b,  PerfOl, 
Perf02b,  Perf02c,  Perf02d,  Perf02e,  OPerf05,  OPerf06,  OPerf()7 


ID 

Question 

Response  range 

Conti  3 

Do  you  separately  cost  and  track  systems  engineering  activities? 

Yes 

No 

Cont14a 

Approximately  what  percentage  of  non-recurring  engineering  (NRE) 
does  systems  engineering  represent? 

Percentages  quantized  as: 

•  <=  5% 

•  <=  10% 

•  <=  15% 

•  <=25% 

.  >  25% 

Cont14b 

Is  the  NRE  percentage  estimated,  or  is  it  a  measured  value? 

•  estimated 

•  measured 

PerfOl 

This  project  creates  and  manages  cost  and  schedule  baselines 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Perf02b 

EVMS  data  are  available  to  decision  makers  in  a  timely  manner  (i.e. 
current  within  2  weeks) 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Perf02c 

The  requirement  to  track  and  report  EVMS  data  is  levied  upon  the 
project’s  suppliers 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Perf02d 

Variance  thresholds  for  CPI  and  SPI  variance  are  defined,  docu¬ 
mented,  and  used  to  determine  when  corrective  action  is  needed 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Perf02e 

EVMS  is  linked  to  the  technical  effort  through  the  WBS  and  the 
IMP/IMS 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

OPerf05 

Does  this  project  track  reports  of  problems  from  fielded  items? 

•  Yes 

•  No 

Scored 
by  the 
number 
of  posi- 

OPerf06 

Does  the  project  conduct  an  engineering  assessment  of  all  field 
trouble  reports? 

•  Yes 

•  No 
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ID 

Question 

Response  range 

OP  erf 07 

The  results  of  this  engineering  assessment  feed  into  ... 

•  operational 
hazard  risk 

assessments 

•  materiel 

readiness  as¬ 
sessments 

•  system  up¬ 
grades  plan¬ 
ning 

•  other 

tive  re¬ 
sponses 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  SEC pmc-  Distribution  of  SECpmc  is  seen  in  F igure  2 1 . 


Maximum  =  3.8 
3rd  Quartile  =  3.2 
Median  =  2.8 
1st  Quartile  =  2.4 
Minimum  =  1.0 
N  =  64 


Figure  21:  Project  Monitoring  and  Control  Capability  ( SECpmc )  Composite  Measure 

Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.3.3. 

5.1. 3.4  Risk  Management  Capability  (SEC rskm) 


The  responding  projects  reported  moderate  to  high  application  of  Risk  Management  best  prac¬ 
tices.  Application  of  Risk  Management  best  practices  was  identified  through  questions  PD  11  and 
PD12. 


ID 

Question 

Response  range 

PDIIa 

This  project  has  a  Risk  Management  process  that  creates  and 
maintains  an  accurate  and  up-to-date  list  of  risks  affecting  the  pro¬ 
ject  (e.g.,  risks  to  cost,  risks  to  schedule,  risks  to  performance) 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PDIIb 

This  project  has  a  Risk  Management  process  that  creates  and 
maintains  up-to-date  documentation  of  risk  mitigation  plans  and 
contingency  plans  for  selected  risks 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 
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ID 

Question 

Response  range 

PDIIc 

This  project  has  a  Risk  Management  process  that  monitors  and 
reports  the  status  of  risk  mitigation  activities  and  resources 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PDIId 

This  project  has  a  Risk  Management  process  that  assesses  risk 
against  achievement  of  an  event-based  schedule 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

PD12 

This  project's  Risk  Management  process  is  integrated  with  program 
decision-making 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  SECrskm ■  Distribution  of  SECrskm  is  seen  in  Figure  22. 


Maximum  =  4.0 
3rd  Quartile  =  3.8 
Median  =  3.0 
1st  Quartile  =  2.8 
Minimum  =  1 .4 
N  =  59 


Figure  22:  Risk  Management  Capability  (SECrskm)  Composite  Measure 

Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.3.4. 

5.1. 3. 5  Requirements  Development  and  Management  Capability  (SEC req) 


The  responding  projects  reported  moderate  to  high  application  of  Requirements  Development  and 
Management  best  practices.  Application  of  Requirements  Development  and  Requirements  Man¬ 
agement  best  practices  were  identified  through  questions  RD01  through  RD1 0 


ID 

Question 

Response  range 

RDOIa 

This  project  maintains  an  up-to-date  and  accurate  listing  of  all  re¬ 
quirements  specified  by  the  customer,  to  include  regulatory,  statu¬ 
tory,  and  certification  requirements 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 
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ID 

Question 

Response  range 

RDOlb 

This  project  maintains  an  up-to-date  and  accurate  listing  of  all  re¬ 
quirements  derived  from  those  specified  by  the  customer 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RD02 

This  project  maintains  up-to-date  and  accurate  documentation 
clearly  reflecting  the  hierarchical  allocation  of  both  customer  and 
derived  requirements  to  each  element  (subsystem,  component, 
etc.)  of  the  system  in  the  configuration  baselines 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RD03a 

This  project  documents  and  maintains  accurate  and  up-to-date 
descriptions  of  operational  concepts  and  their  associated  scenarios 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RD03b 

This  project  documents  and  maintains  accurate  and  up-to-date 
descriptions  of  use  cases  (or  their  equivalent) 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RD03c 

This  project  documents  and  maintains  accurate  and  up-to-date 
descriptions  of  product  installation,  maintenance  and  support  con¬ 
cepts 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RD04 

This  project  has  documented  criteria  for  identifying  authorized  re¬ 
quirements  providers  to  avoid  requirements  creep  and  volatility 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RD05 

This  project  has  documented  criteria  (e.g.,  cost  impact,  schedule 
impact,  authorization  of  source,  contract  scope,  requirement  qual¬ 
ity)  for  evaluation  and  acceptance  of  requirements 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RD06 

The  requirements  for  this  project  are  approved  in  a  formal  and  do¬ 
cumented  manner  by  relevant  stakeholders 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RD07 

This  project  performs  and  documents  requirements  impact  as¬ 
sessments  for  proposed  requirements  changes 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RD08 

This  project  develops  and  documents  project  requirements  based 
upon  stakeholder  needs,  expectations,  and  constraints 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RD09 

This  project  has  an  accurate  and  up-to-date  requirements  tracking 
system 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RDlOa 

For  this  project,  the  requirements  documents  are  managed  under  a 
configuration  control  process 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 
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ID 

Question 

Response  range 

RDlOb 

For  this  project,  the  requirements  documents  are  accessible  to  all 
relevant  project  staff 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  SECreq.  Distribution  of  SECreq  is  seen  in  Figure  23. 


Maximum  =  4.0 
3rd  Quartile  =  3.4 
Median  =  3.0 
1st  Quartile  =  2.8 
Minimum  =  2.2 
N  =  58 


Figure  23:  Requirements  Development  and  Management  Capability  ( SECreq )  Composite  Measure 

Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.3.5. 

5.1. 3. 6  Trade  Studies  Capability  (SECt/mde) 


The  responding  projects  reported  moderate  to  high  application  of  Trade  Studies  best  practices. 
Application  of  Trade  Study  best  practices  was  identified  through  questions  RD1 1  through  RD13 


ID 

Question 

Response  range 

RD11 

Stakeholders  impacted  by  trade  studies  are  involved  in  the  devel¬ 
opment  and  performance  of  those  trade  studies 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RD12 

This  project  performs  and  documents  trade  studies  between  alter¬ 
nate  solutions  based  upon  definitive  and  documented  selection 
criteria 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RD13 

Documentation  of  trade  studies  is  maintained  in  a  defined  reposi¬ 
tory  and  is  accessible  to  all  relevant  project  staff 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 
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Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  SECtrade.  Distribution  of  SECTRADe  is  seen  in  Figure  24. 


Maximum  =  4.0 
3rd  Quartile  =  3.7 
Median  =  3.0 
1st  Quartile  =  2.3 
Minimum  =  1 .0 
N  =  58 


Figure  24:  Trade  Study  Capability  (SECTRade)  Composite  Measure 

Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.3.6. 

5.1. 3.7  Architecture  Capability  (SEC arch) 


The  responding  projects  reported  moderate  to  high  application  of  Architecture  best  practices. 
Application  of  Architecture  best  practices  was  identified  through  questions  IF01  through  IF04. 


ID 

Question 

Response  range 

IF01 

This  project  maintains  accurate  and  up-to-date  descriptions  (e.g. 
interface  control  documents,  models,  etc.)  defining  interfaces  in 
detail 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

IF02 

Interface  definition  descriptions  are  maintained  in  a  designated 
location,  under  configuration  management,  and  accessible  to  all 
who  need  them 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

IF03a 

For  this  project,  the  product  high-level  structure  is  documented, 
kept  up  to  date,  and  managed  under  configuration  control 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

IF03b 

For  this  project,  the  product  high-level  structure  is  documented 
using  multiple  views  (e.g.  functional  views,  module  views,  etc. 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

IF03c 

For  this  project,  the  product  high-level  structure  is  accessible  to  all 
relevant  project  personnel 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 
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ID 

Question 

Response  range 

IF04 

This  project  has  defined  and  documented  guidelines  for  choosing 
COTS  product  components 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  SECarch-  Distribution  of  SEC  arch  is  seen  in  Figure  25. 


Maximum  =  4.0 
3rd  Quartile  =  3.5 
Median  =  2.8 
1st  Quartile  =  2.6 
Minimum  =  2.0 
N  =  57 


Figure  25:  Product  Architecture  Capability  ( SECarch  )  Composite  Measure 

Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.3.7. 

5.1. 3. 8  Technical  Solution  Capability  (SECrs) 

The  responding  projects  reported  moderate  to  high  application  of  Technical  Solution  best  prac¬ 
tices.  Application  of  Technical  Solution  best  practices  was  identified  through  questions  RD11 
through  RD13  and  IF01  through  IF04.  SECts  is  actually  an  aggregate  of  Trade  Study  Capability 
(SECtrade)  and  Architecture  Capability  (SECarch) 


ID 

Question 

Response  Range 

RD11 

Stakeholders  impacted  by  trade  studies  are  involved  in  the  devel¬ 
opment  and  performance  of  those  trade  studies 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RD12 

This  project  performs  and  documents  trade  studies  between  alter¬ 
nate  solutions  based  upon  definitive  and  documented  selection 
criteria 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

RD13 

Documentation  of  trade  studies  is  maintained  in  a  defined  reposi¬ 
tory  and  is  accessible  to  all  relevant  project  staff 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 
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ID 

Question 

Response  Range 

IF01 

This  project  maintains  accurate  and  up-to-date  descriptions  (e.g. 
interface  control  documents,  models,  etc.)  defining  interfaces  in 
detail 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

IF02 

Interface  definition  descriptions  are  maintained  in  a  designated 
location,  under  configuration  management,  and  accessible  to  all 
who  need  them 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

IF03a 

For  this  project,  the  product  high-level  structure  is  documented, 
kept  up  to  date,  and  managed  under  configuration  control 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

IF03b 

For  this  project,  the  product  high-level  structure  is  documented 
using  multiple  views  (e.g.  functional  views,  module  views,  etc.) 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

IF03c 

For  this  project,  the  product  high-level  structure  is  accessible  to  all 
relevant  project  personnel 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

IF04 

This  project  has  defined  and  documented  guidelines  for  choosing 
COTS  product  components 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  SECts .  Distribution  of  SECts  is  seen  in  Figure  26. 


1  h 


Maximum  =  4.0 
3rd  Quartile  =  3.3 
Median  =  2.9 
1st  Quartile  =  2.6 
Minimum  =  2.1 
N  =  57 


Figure  26:  Technical  Solution  (SECts)  Composite  Measure 

Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.3.8. 
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5.1. 3.9  Product  Integration  Capability  (SEC pi) 


The  responding  projects  reported  moderate  to  high  application  of  Product  Integration  best  prac¬ 
tices.  Application  of  Product  Integration  best  practices  was  identified  through  question  IF05. 


ID 

Question 

Response  range 

IF05 

This  project  has  accurate  and  up-to-date  documents  defining  its 
product  integration  process,  plans,  criteria,  etc.  throughout  the  life 
cycle 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

After  normalization,  this  response  was  used  to  calculate  SECpi.  Distribution  of  SECpi  is  seen  in 
Figure  27. 


Maximum  =  4.0 
3rd  Quartile  =  3.0 
Median  =  3.0 
1st  Quartile  =  2.0 
Minimum  =  2.0 
N  =  57 


Figure  27:  Product  Integration  Capability  (SECpj )  Measure 

A  distribution  of  the  individual  responses  to  this  question  is  found  in  APPENDIX  D,  Section 
D.3.9. 

5.1.3.10  Verification  Capability  (SECv'er) 


The  responding  projects  reported  moderate  to  high  application  of  Verification  best  practices.  Ap¬ 
plication  of  Verification  best  practices  was  identified  through  questions  V&V01  through  V&V03. 


ID 

Question 

Response  range 

V&VOIa 

This  project  has  accurate  and  up-to-date  documents  defining  the 
procedures  used  for  the  test  and  verification  of  systems  and  system 
elements 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

V&VOlb 

This  project  has  accurate  and  up-to-date  documents  defining  ac¬ 
ceptance  criteria  used  for  the  verification  of  systems  and  system 
elements 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 
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ID 

Question 

Response  range 

V&V02a 

This  project  has  a  documented  and  practiced  review  (e.g.  peer 
reviews,  design  reviews,  etc.)  process  that  defines  entry  and  exit 
criteria  for  work  products 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

V&V02b 

This  project  has  a  documented  and  practiced  review  (e.g.  peer 
reviews,  design  reviews,  etc.)  process  that  includes  training  re¬ 
quirements  for  the  reviewers 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

V&V02e 

This  project  has  a  documented  and  practiced  review  (e.g.  peer 
reviews,  design  reviews,  etc.)  process  that  addresses  identified 
risks  and  risk  mitigation  activities  during  reviews 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

V&V02f 

This  project  has  a  documented  and  practiced  review  (e.g.  peer 
reviews,  design  reviews,  etc.)  process  that  examines  completeness 
of  configuration  baselines 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

V&V03 

This  project  conducts  non-advocate  reviews  (e.g.  reviews  by  quali¬ 
fied  personnel  with  no  connection  to  or  stake  in  the  project)  and 
documents  results,  issues,  action  items,  risks,  and  risk  mitigations 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

V&V02C 

This  project  has  a  documented  and  practiced  review  (e.g.  peer 
reviews,  design  reviews,  etc.)  process  that  defines  criteria  for  the 
selection  of  work  products  (e.g.,  requirements  documents,  test 
plans,  system  design  documents,  etc.)  for  review 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

V&V02d 

This  project  has  a  documented  and  practiced  review  (e.g.  peer 
reviews,  design  reviews,  etc.)  process  that  tracks  action  items  to 
closure 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  SECver ■  Distribution  of  SECver  is  seen  in  Figure  28. 


Maximum  =  4.0 
3rd  Quartile  =  3.4 
Median  =  3.0 
1st  Quartile  =  2.6 
Minimum  =  2.2 
N  =  58 


Figure  28:  Verification  Capability  (SECver)  Composite  Measure 
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Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.3.10. 

5.1.3.11  Validation  Capability  (SECwu.) 


The  responding  projects  reported  moderate  to  high  application  of  Validation  best  practices.  Ap¬ 
plication  of  Validation  best  practices  was  identified  through  questions  V&V04  and  V&V05. 


ID 

Question 

Response  Rate 

V&V04a 

This  project  has  accurate  and  up-to-date  documents  defining  the 
procedures  used  for  the  validation  of  systems  and  system  elements 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

V&V04b 

This  project  has  accurate  and  up-to-date  documents  defining  ac¬ 
ceptance  criteria  used  for  the  validation  of  systems  and  system 
elements 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

V&V05 

This  project  maintains  a  listing  of  items  managed  under  configura¬ 
tion  control 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  SECVaL.  Distribution  of  SECyu  is  seen  in  Figure  29. 


Maximum  =  4.0 
3rd  Quartile  =  3.7 
Median  =  3.0 
1st  Quartile  =  2.7 
Minimum  =  1.7 
N  =  58 


Figure  29:  Validation  Capability  (SECval)  Composite  Measure 

Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.3.11. 

5.1.3.12  Configuration  Management  Capability  (SEC cm) 

The  responding  projects  reported  high  application  of  Configuration  Management  best  practices. 
Application  of  Configuration  Management  best  practices  was  identified  through  questions  V&  V06 
and  V&V07. 
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ID 

Question 

Response  Range 

V&V06 

This  project  has  a  configuration  management  system  that  charters 
a  Change  Control  Board  to  disposition  change  requests 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

V&V07 

This  project  maintains  records  of  requested  and  implemented 
changes  to  configuration-managed  items 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

V&V08 

This  project  creates  and  manages  configuration  baselines  (e.g., 
functional,  allocated,  product) 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  SECcm •  Distribution  of  SECCm  is  seen  in  Figure  30. 


Maximum  =  4.0 
3rd  Quartile  =  4.0 
Median  =  3.6 
1st  Quartile  =  3.0 
Minimum  =  2.0 
N  =  58 


Figure  30:  Configuration  Management  Capability  (SECcm)  Composite  Measure 

Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.3.12. 

5.1.3.13  Overall  Systems  Engineering  Capability  (SEC) 

The  capability  subcategories  of  5. 1.3.1  through  5.1.3.12  may  be  combined  to  produce  a  measure 
of  overall  Systems  Engineering  Capability  (SEC).  After  normalization,  the  results  of  each  sub¬ 
category  were  linearly  combined  to  create  SEC.  Distribution  of  SEC  is  seen  in  Figure  3 1 .  The 
responding  projects  reported  moderate  to  high  Systems  Engineering  Capability. 
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Maximum  =  3.9 
3rd  Quartile  =  3.3 
Median  =  3.0 
1st  Quartile  =  2.7 
Minimum  =  2.1 
N  =  63 


Figure  31:  Overall  Systems  Engineering  Capability  (SEC)  Composite  Measure 

5.1.4  Acquirer  Capability  (AC) 

Because  the  survey  respondents  are  the  project  suppliers,  and  not  the  project  acquirers,  any  infor¬ 
mation  gathered  regarding  the  acquirers  is  second-hand  information;  that  is,  it  is  an  evaluation  of 
the  acquirer  from  the  perspective  of  the  supplier.  Nevertheless,  there  are  a  few  parameters  that  can 
be  measured  to  imply  Acquirer  Capability;  parameters  such  as 

•  acquirer’s  participation  on  Integrated  Project  Teams  (IPTs) 

•  acquirer’s  provision  of  a  Systems  Engineering  Plan  (SEP) 

•  quality  of  system  requirements 

•  completeness  of  system  requirements 

•  stability  of  system  requirements 

Although  this  survey  was  not  specifically  designed  to  assess  the  capabilities  of  the  acquirers,  these 
parameters  can  be  combined  to  develop  a  rudimentary  measure  of  Acquirer  Capability  (AC). 


The  responding  projects  reported  moderate  to  high  Acquirer  Capability.  The  acquirer’s  capability 
was  identified  through  questions  Proj05,  ProjlOa ,  ProjlOb,  PD10,  Perf2a,  Conti  1,  and  Contl2. 


ID 

Question 

Response  Range 

Proj05 

Both  the  supplier  and  the  acquirer  actively  participate  in  IPTs 

•  Strongly  Disagree 

•  Disagree 

•  Agree 

•  Strongly  Agree 

ProjlOa 

The  requirements  for  this  project  are  well-defined 

•  Strongly  Disagree 

•  Disagree 

•  Agree 

•  Strongly  Agree 

ProjlOb 

The  requirements  for  this  project  have  not  changed  significantly 
throughout  the  life  of  the  project  to-date 

•  Strongly  Disagree 

•  Disagree 

•  Agree 

•  Strongly  Agree 
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ID 

Question 

Response  Range 

PD10 

The  acquirer  has  provided  this  project  with  a  Systems  Engineering 
Plan 

•  Strongly  Disagree 

•  Disagree 

•  Agree 

•  Strongly  Agree 

Perf2a 

Your  customer  requires  that  you  supply  EVMS  data 

•  Strongly  Disagree 

•  Disagree 

•  Agree 

•  Strongly  Agree 

Conti  1 

What  percentage  of  the  customer  technical  requirements  were 
marked  “To  Be  Determined”  at  time  of  contract  award? 

•  <1% 

•  1-5% 

•  5-20% 

•  >20% 

Cont12 

What  percentage  of  the  customer’s  technical  requirements  are  cur¬ 
rently  marked  “To  Be  Determined”? 

•  <1% 

•  1-5% 

•  5-20% 

•  >20% 

Using  the  process  described  in  Section  5,  the  responses  to  these  questions  were  combined  to  cre¬ 
ate  AC.  Distribution  of  AC  is  seen  in  Figure  32. 


Maximum  =  4.0 
3rd  Quartile  =  3.1 
Median  =  2.8 
1st  Quartile  =  2.4 
Minimum  =  1 .5 
N  =  64 


Figure  32:  Acquirer  Capability  (AC)  Composite  Measure 

Distributions  of  the  individual  responses  to  these  questions  are  found  in  APPENDIX  D,  Section 
D.4. 

5.1.5  Project  Performance 

As  noted  earlier,  project  cost,  schedule,  and  scope  are  in  opposition  in  a  typical  project.  An  at¬ 
tempt  to  improve  one  of  these  factors  is  often  met  with  deterioration  of  the  other  two.  In  most  cas¬ 
es,  one  of  these  factors  is  given  priority  over  the  other  two.  Some  examples  follow. 

•  For  a  project  with  strong  financial  constraints,  meeting  project  cost  goals  may  be  a  priority. 
This  may  necessitate  schedule  delays  due  to  limits  on  resources.  Scope  reductions  may  also 
be  applied  to  reduce  the  project  effort. 

•  For  a  project  with  strong  schedule  constraints  (e.g.,  a  weapons  system  needed  in  the  field 
NOW!),  achieving  delivery  schedules  may  be  a  priority.  This  may  necessitate  additional  costs 
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arising  from  more  resources  being  applied  to  accelerate  the  program.  Scope  reductions  may 
also  be  applied  to  eliminate  or  shorten  project  tasks. 

•  For  a  project  with  strong  scope  satisfaction  constraints  (e.g.,  a  mission-  or  safety-critical  sys¬ 
tem),  achieving  the  specified  scope  may  be  a  priority.  This  may  necessitate  additional  costs 
arising  from  more  resources  being  applied  to  achieve  desired  performance.  Schedule  slippage 
may  occur  as  effort  expands  to  address  scope  shortfalls. 

The  result  is  that  Project  Performance  cannot  be  measured  by  cost  compliance,  schedule  compli¬ 
ance,  or  scope  compliance  alone.  All  three  must  be  considered. 

5.1. 5.1  Cost  Performance  (Perfc) 

The  project’s  cost  performance  was  identified  through  questions  ContOl,  Cont03,  Perf04a, 

Perf05a,  and  Per/06. 


ID 

Question 

ContOl 

What  is  the  current  total  contract  value  of  this  project? 

Cont03 

What  was  the  initial  contract  value  of  this  project? 

Perf04a 

What  is  the  current  estimated  cost  at  completion  for  this  project? 

Perf05a 

What  is  the  projected  cost  variance  at  completion  for  the  current 

contract  baseline? 

Perf06 

What  is  the  current  cumulative  (or  final)  EVMS  Cost  Performance 
Index  (CPI)  for  this  project? 

Calculation  of  a  measure  of  cost  performance  was  somewhat  more  difficult  than  calculation  of 
supplier  capabilities  as  described  in  Section  5.1.  The  data  upon  which  to  fonn  an  evaluation  in¬ 
cluded 

•  initial  contract  value  (CVi) 

•  current  contract  value  (CVc) 

•  current  estimated  cost-at-completion  (ECACc) 

•  current  estimated  cost  variance  at  completion  (EVACc) 

•  EVMS  cost  performance  index  (CPIc) 

ECACc  and  EVACc  were  analyzed  to  identify  the  percent-cost  variance  of  the  project.  CPI  was 
separately  evaluated.  Projects  were  then  graded  on  a  scale  of  1  to  4  as  follows: 

4  =  under  budget  3  =  on  budget  (0  to  2%  over  budget) 

2=  2  to  1 0%  over  budget  1=  >10%  over  budget 

5.1. 5.2  Schedule  (duration)  Performance  ( Perfo ) 


The  project’s  schedule  (i.e.,  duration)  performance  was  identified  through  questions  Cont02, 
Cont04 ,  Perf04b,  Perf05b,  Perf07,  OPer/03,  and  Oper/04 


ID 

Question 

Response  range 

Cont02 

What  is  the  current  total  planned  duration  of  this  project? 
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ID 

Question 

Response  range 

Cont04 

What  was  the  initial  total  planned  duration  of  this  project? 

Perf04b 

What  is  the  current  estimated  total  duration  for  this  project? 

Perf05b 

What  is  the  projected  schedule  variance  at  completion  for  the  cur¬ 
rent  contract  baseline? 

PerfOJ 

What  is  the  current  cumulative  (or  final)  EVMS  Schedule  Perform¬ 
ance  Index  (SPI)  for  this  project? 

OPerf03 

Overall,  this  project  is  performing  per  the  schedule  established  in 
the  current  IMS  approved  by  the  acquirer 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

Operf04 

The  schedule  of  this  project’s  critical  path,  when  compared  to  the 
current  IMS  approved  by  the  acquirer  is  ... 

•  >6  months  late 

•  3-6  months  late 

•  1-3  months  late 

•  within  +/- 1  month 

•  1-3  months  early 

•  3-6  months  early 

•  >6  months  early 

Calculation  of  a  measure  of  schedule  performance  was  similar  to  that  for  a  measure  of  cost  per¬ 
formance.  The  data  upon  which  to  fonn  an  evaluation  included 

•  current  total  planned  project  duration  (PDc) 

•  initial  total  planned  project  duration  (PDi) 

•  current  estimated  total  duration  for  this  project  (EDc) 

•  projected  schedule  variance  at  completion  for  the  current  contract  baseline  (DV) 

•  current  cumulative  (or  final)  EVMS  schedule  performance  index  (SPI) 

•  EVMS  update  frequency 

•  current  completion  status  of  this  project 

EDc  and  DVc  were  analyzed  to  identify  the  percent-schedule  variance  of  the  project.  SPI  was 
separately  evaluated.  Projects  were  then  graded  on  a  scale  of  1  to  4  as  follows: 

4  =  early  3  =  on  schedule  (0  to  2%  late) 

2  =  2  to  10%  late  1  =  >10%  late 


5.1. 5. 3  Scope  Satisfaction  Performance  (Perfs) 


The  project’s  scope  performance  was  identified  through  question  OPerf02 


ID 

Question 

Response  range 

OPerf02 

Requirements  are  being  satisfied  and  remain  on  track  to  be  satis¬ 
fied  in  the  product  releases  as  originally  planned;  they  are  not  being 
deleted  or  deferred  to  later  releases 

•  strongly  disagree 

•  disagree 

•  agree 

•  strongly  agree 

54  |  CMU/SEI-2007-SR-014 


ND1A 


5.1. 5.4  Total  Performance  ( Perf ) 


The  measure  Perf  represents  this  total  Project  Performance  and  is  calculated  as  the  combination 
of  Perfc,  PerfD,  and  Perfs.  Distribution  of  Perf  is  seen  in  Figure  33.  The  responding  projects  re¬ 
ported  moderate  to  high  Project  Performance 


Maximum  =  4.0 
3rd  Quartile  =  3.1 
Median  =  2.75 
1st  Quartile  =  2.3 
Minimum  =  1.7 
N  =  46 


Figure  33:  Total  Project  Performance  ( Perf) 

For  the  purposes  of  the  remaining  analysis,  the  respondents  were  grouped  into  one  of  three  cate¬ 
gories: 


•  Best  Performance  Perf>  3.0 

•  Moderate  Performance  2.5  <  Perf  <3.0 

•  Lower  Performance  Perf<  2.5 

This  trichotomy  placed  approximately  equal  numbers  of  respondents  within  each  category. 

We  must  stress  the  relative  nature  of  these  categories.  These  Project  Performance  categories  do 
not  range  from  worst  possible  performance  score  to  the  best  possible  performance  score.  Instead, 
they  range  from  the  lowest  performance  score  achieved  by  any  of  projects  in  the  survey  sample  to 
the  highest  performance  score  that  was  achieved. 

5.2  RELATIONSHIPS  BETWEEN  SEC,  PC,  PE,  AC,  AND  PERF 

The  objective  of  this  survey  is  to  identify  relationships  between  Systems  Engineering  Capability 
and  Project  Performance.  Given  that  our  hypothesis  is 

Perf =f  (PC,  SEC,  AC,  PE) 

We  will  accomplish  this,  by  examining  the  following  relationships: 

•  Project  Performance  (Perf)  versus  Systems  Engineering  Capability  ( SEC) 
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•  Project  Performance  ( Perf)  versus  Project  Challenge  (PE) 

•  Project  Performance  ( Perf)  versus  Project  Environment  ( PE) 

•  Project  Performance  (Perf)  versus  Acquirer  Capability  ( AC) 

We  will  also  examine  the  relationship  between  Project  Performance  ( Perf)  and  Systems  Engineer¬ 
ing  Capability  (SEC)  as  moderated  by 

•  Project  Challenge 

•  Project  Environment 

•  Acquirer  Capability 

Relationships  between  driving  factors  (that  is,  Systems  Engineering  Capability  subcategories 
(SECxxx),  Project  Environment  Factors,  Project  Challenge)  and  Performance  (Perf)  are  illustrated 
using  a  mosaic  graph.  The  mosaic  graph  provides  an  intuitive  means  of  examining  the  statistical 
relationship  between  a  dependent  variable  (Project  Performance,  depicted  on  the  vertical  axis)  and 
an  independent  variable  (such  as  a  Systems  Engineering  Capability,  depicted  on  the  horizontal 
axis). 

As  an  example,  Figure  34  shows  an  illustration  of  the  relationship  between  two  survey  variables: 
Project  Planning  capability  and  Project  Performance.  As  noted  in  Section  5. 1.3. 2,  the  responses  to 
a  number  of  survey  questions  are  processed  to  obtain  a  quantitative  assessment  of  the  supplier’s 
Project  Planning  capabilities.  Similarly,  in  Section  5.1.4,  other  questions  are  processed  to  obtain  a 
quantitative  assessment  of  the  supplier’s  overall  performance  on  the  project.  In  constructing  this 
graphic,  we  first  establish  thresholds  that  enable  us  to  define  three  levels  of  Project  Planning  ca¬ 
pability 

•  higher  Project  Planning  capability 

•  moderate  Project  Planning  capability 

•  lower  Project  Planning  capability 

The  distribution  of  survey  responses  within  these  categories  is  one  of  the  criteria  used  in  establish¬ 
ing  these  thresholds.  We  then  partition  the  data  set,  binning  the  survey  responses  per  the  thresh¬ 
olds. 
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Projects  exhibiting  the  Projects  exhibiting  an  Projects  exhibiting  the 

LOWEST  level  of  INTERMEDIATE  level  HIGHEST  level  of 


Project  Planning  of  Project  Planning  Project  Planning 

capability  capability  capability 


Figure  34:  Mosaic  Chart  Key 

Likewise,  we  establish  thresholds  that  enable  us  to  define  three  levels  of  Project  Performance: 

•  high  Project  Performance 

•  intermediate  Project  Performance 

•  low  Project  Performance 

For  each  of  the  Project  Planning  capability  bins,  we  can  then  illustrate  the  distribution  of  Project 
Performance  as  a  stacked  column  graph. 

The  mosaic  graph  provides  additional  information  found  in  the  width  of  each  of  its  columns.  This 
width  is  proportional  to  the  quantity  of  responses  within  that  bin.  Finally,  the  column  graph  on  the 
right  shows  the  distribution  of  Project  Performance  responses  for  the  entire  sample  (that  is,  for  all 
Project  Planning  bins  combined). 
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A  measure  of  association  and  statistical  test  are  also  included. 


•  Gamma  is  a  measure  of  association  that  expresses  the  strength  of  relationship  between  two 
ordinal  variables.  A  clear,  simple  description  of  Goodman  and  Kruskal's  gamma  may  be 
found  in  Linton  Freeman’ s  Elementary  Applied  Statistics  [Freeman  1965].  Values  of  gamma 
are  based  on  the  difference  between  concordant  (P)  and  discordant  (Q)  paired  comparisons  of 
the  two  variables.  It  is  computed  as  (P-Q)/(P+Q),  i.e.,  the  excess  of  concordant  pairs  as  a  per¬ 
centage  of  all  pairs  ignoring  ties.  Similar  to  Pearson’s  product  moment  relationship  coeffi¬ 
cient  (r),  gamma  varies  from  +1  to  -1,  with 

values  near  -1  indicating  a  strong  opposing  relationship 

values  near  0  indicating  a  weak  or  no  relationship  (statistical  independence) 

values  near  +1  indicating  a  strong  supporting  relationship 

Gamma  is  a  Proportional  Reduction  in  Error  (PRE)  statistic,  so  understanding  its  value  is  in¬ 
tuitively  straightforward.  Conceptually  similar  to  Pearson’s  r2  for  interval  or  ratio  data,  a 
gamma  value  can  be  interpreted  as  the  proportion  of  paired  comparisons  where  knowing  the 
rank  order  of  one  variable  allows  one  to  predict  accurately  the  rank  order  of  the  other  vari¬ 
able. 

Notionally,  gamma  values  of  less  than  0.2  may  be  considered  as  weak,  and  values  around  0.3 
can  be  thought  of  as  moderately  strong.  Gamma  values  in  the  neighborhood  of  0.5  can  be 
characterized  as  strong,  while  values  over  0.6  are  quite  high  for  categoric  survey  data  such  as 
those  in  this  report. 

•  “p”  is  generally  interpreted  as  the  probability  that  one  would  observe  a  statistical  relationship 
in  a  sample  of  data  by  chance  alone.  By  convention,  values  of  p  <  0.05  ox  p  <  0.01  typically 
are  used  as  a  basis  for  rejecting  the  null  hypothesis,  i.e.,  having  confidence  that  the  relation¬ 
ship  is  not  specious. 

Because  of  the  small  number  of  cases  in  the  present  survey,  the  p  values  for  some  of  the 
weaker  relationships  are  greater  than  0.05.  However,  the  percentage  differences  and  related 
gamma  values  themselves  are  more  meaningful  for  understanding  the  results  than  are  the  p 
values  per  se. 

Given  the  way  in  which  the  sample  was  drawn,  we  cannot  generalize  our  univariate  findings 
to  the  larger  population  of  DoD  programs;  however,  there  is  sufficient  variation  to  analyze  the 
relationships  among  the  variables.  It  is  those  relationships  that  allow  us  to  address  the  validity 
of  assertions  about  the  effects  on  program  performance  of  Systems  Engineering  activities  un¬ 
der  varying  circumstances. 

With  this  understanding,  interpretation  of  the  mosaic  graph  is  straightforward.  Figure  34  tells  us: 

•  Approximately  25%  (estimated  from  the  width  of  the  first  column)  of  the  survey  respondents 
exhibit  low  Project  Planning  Capability  on  their  projects.  Within  this  group: 

•  60%  of  the  projects  show  low  Project  Performance 

•  25%  of  the  projects  show  intermediate  Project  Performance,  and 

•  15%  of  the  projects  show  high  Project  Performance 
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•  Approximately  35%  (estimated  from  the  width  of  the  second  column)  of  the  survey  respon¬ 
dents  exhibit  intermediate  Project  Planning  Capability  on  their  projects.  Within  this  group: 

•  20%  of  the  projects  show  low  Project  Performance 

•  50%  of  the  projects  show  intermediate  Project  Performance,  and 

•  30%  of  the  projects  show  high  Project  Performance 

•  Approximately  40%  (estimated  from  the  width  of  the  third  column)  of  the  survey  respondents 
exhibit  high  Project  Planning  Capability  on  their  projects.  Within  this  group: 

•  10%  of  the  projects  show  low  Project  Performance 

•  25%  of  the  projects  show  intermediate  Project  Performance,  and 

•  65%  of  the  projects  show  high  Project  Performance 

•  Gamma  =  0.64  describes  the  strong  supporting  relationship  between  Project  Planning  capabil¬ 
ity  and  Project  Performance,  while  p  =  0.03  indicates  that  the  likelihood  of  a  relationship  of 
this  magnitude  happening  by  chance  alone  is  only  3%. 

Clearly,  in  this  hypothetical  case,  better  Project  Planning  capability  is  related  to  better  Project  Per¬ 
formance. 

Note  that  the  choice  of  bins  is  not  necessarily  limited  to  three.  In  principle,  the  data  could  be  parti¬ 
tioned  into  two,  four,  or  any  number  of  bins.  We  use  three  categories  in  this  Special  Report  be¬ 
cause  the  relatively  small  number  of  projects  that  participated  in  the  survey  limits  the  confidence 
that  one  can  have  in  the  differences  among  categories.  The  number  of  comparisons  cannot  mean¬ 
ingfully  approach  the  number  of  cases. 

It  should  be  stressed  that,  unlike  the  distribution  graphs  presented  in  Section  5.1,  the  mosaic 
charts  describe  relative  rather  than  absolute  differences.  The  Project  Performance  categories  on 
the  vertical  axis  do  not  range  from  worst  possible  performance  score  to  the  best  possible  perform¬ 
ance  score.  Instead,  they  range  from  the  lowest  performance  score  achieved  by  any  of  projects  in 
the  survey  sample  to  the  highest  performance  score  that  was  achieved.  Thus,  on  an  absolute  scale 
of  1  (worst  possible  performance)  to  4  (best  possible  performance),  if  all  of  the  respondent’s  had 
indicated  that  their  projects  were  performing  relatively  well  and  fell  into  the  range  from  2  to  4,  the 
mosaic  graph  might  consider  those  scoring  from  2  to  2.7  as  “Lower  Performance,”  those  scoring 
from  2.8  to  3.2  as  “Moderate  Performance,”  and  those  scoring  from  3.3  to  4  as  “Best  Perform¬ 
ance”.  The  same  is  true  for  the  Capability  measure  of  the  horizontal  axis.  It  again  is  relative  in 
nature,  ranging  from  the  lowest  capability  reported  to  the  highest. 

The  relationships  discussed  throughout  Section  5.2  are  also  summarized  in  Table  7,  found  in  Sec¬ 
tion  7. 

5.2.1  Relationships  between  Project  Challenge  (PC)  and  Project  Performance  (Perf) 

Project  Challenge  may  have  a  significant  impact  upon  Project  Performance.  As  expected  the  Pro¬ 
ject  Challenge  measure  described  in  Section  5.1.1  showed  a  moderately  strong  negative  statistical 
relationship  with  the  Project  Performance  measure  defined  in  Section  5. 1.5. 4,  as  shown  in  Figure 
35. 


NATIONAL  DEFENSE  INDUSTRIAL  ASSOCIATION 


SOFTWARE  ENGINEERING  INSTITUTE  |  59 


Lower  Moderate  Higher 

Challenge  Challenge  Challenge 

(x  <1 .85)  (1 .85<x<2.05)  (x  >2.05) 

N  =  18  N  =  12  N  =  16 


Best  Performance 
(x>  3.0) 

Moderate  Per¬ 
formance 
(2.5  <  x  <  3.0) 

Lower  Performance 
(x  <  2.5) 


Gamma  =  -0.31 
p  =  0.05 


Figure  35:  Relationship  Between  Project  Challenge  and  Performance  ( Perf  Versus  PC ) 

Project  Challenge  varied  across  the  project  sample  with  18  projects  having  Lower  Project  Chal¬ 
lenge,  12  having  a  Moderate  Project  Challenge,  and  16  having  Higher  Project  Challenge.  Fifty 
percent  of  the  projects  with  Lower  Project  Challenge  exhibited  Best  Performance,  while  only  25% 
of  the  projects  with  Higher  Project  Challenge  exhibited  Best  Performance.  Similarly,  although 
less  consistent  over  the  range  of  Project  Challenge,  only  22%  of  projects  with  Lower  Project 
Challenge  exhibited  Lower  Performance,  while  only  38%  of  projects  with  Higher  Project  Chal¬ 
lenge  exhibited  Lower  Performance. 

A  Gamma  value  of  -0.3 1  confirms  that  there  is  a  moderately  strong  negative  relationship  between 
Project  Perfonnance  and  the  elements  of  Process  Challenge  addressed  in  this  survey;  i.e.,  per¬ 
formance  degrades  as  the  projects  become  more  difficult.  A p  value  of  0.05  indicates  that  there  is 
a  5%  probability  that  this  type  of  relationship  could  spontaneously  occur  by  chance  alone. 


5.2.2  Relationships  between  Project  Environment  ( PE)  and  Project  Performance  (Perf) 


Factors  other  than  Systems  Engineering  Capability  may  impact  Project  Performance.  To  identify 
these  impacts,  we  will  examine  the  following  relationships: 


Customer  category  versus  Performance 
Acquiring  organization  versus  Performance 
Position  in  Systems  Hierarchy  versus  Performance 
Subcontracted  percentage  versus  Performance 
Systems  Engineering  Content  versus  Perfonnance 
CMMI-based  process  management  versus  Performance 
Prior  Experience  versus  Performance 
Process  Improvement  versus  Performance 


( 

ProdOl 

versus 

Perf  ) 
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Prod02 

versus 

Perf  ) 
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Prod04 

versus 

Perf  ) 

( 

Cont08 

versus 
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versus 
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SECcmmi 

versus 

Perf  ) 
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SEC  exp 

versus 
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( 
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versus 
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Most  of  the  Project  Environment  (PE)  measures  in  the  relationships  with  Project  Performance 
(Perf)  that  follow  are  not  easily  amenable  to  sound  summary  statistical  analysis.  There  either  are 
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not  enough  projects  in  each  of  the  possible  response  categories  or  the  range  of  answers  the  pro¬ 
jects  gave  cannot  be  classified  evenly  enough  into  homogeneous  categories.  Some  pertinent  per¬ 
centage  differences  do  exist.  However,  the  summary  relationships  remain  obscure.  Hence,  we 
have  refrained  from  presenting  Gamma  measures  of  association  and  statistical  tests  in  this  section. 

In  question  ProdOl,  customers  were  categorized  as  Government,  Prime  Contractor,  or  Subcon¬ 
tractor,  as  noted  in  Section  5.1.2. 
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Figure  36:  Relationship  Between  Customer  Category  and  Performance  ( Perf  Versus  ProdOl ) 

As  shown  in  Figure  36,  there  was  some  variation  based  upon  the  customer  to  whom  the  product 
was  delivered.  U.S.  government  organizations  were  the  customer  for  32  projects  (i.e.,  the  project 
was  being  executed  by  the  Prime  Contractor).  Of  these  projects,  only  22%  exhibited  Best  Per¬ 
formance.  The  Prime  Contractor  was  the  customer  for  13  projects  (i.e.,  the  project  was  being  exe¬ 
cuted  by  a  first-tier  subcontractor).  Of  these  projects,  the  percentage  exhibiting  Best  Performance 
increased  to  38%.  Projects  where  the  customer  was  a  subcontractor  (i.e.,  where  the  project  was 
being  executed  by  a  lower  tier  contractor),  are  not  shown  here.  The  number  of  projects  in  this  cat¬ 
egory  was  insufficient  for  meaningful  analysis  and  too  small  to  honor  our  promise  of  non¬ 
disclosure  made  to  the  survey  participants. 

Various  hypotheses  explaining  this  result  could  be  made.  In  general,  one  would  expect  the  Prime 
Contractor’s  projects  to  be  larger  and  more  complex  than  the  Subcontractor’s  projects.  This  in¬ 
creased  size  and  complexity  could  be  a  factor  in  the  lower  performance.  An  alternate  interpreta¬ 
tion  is  that  perhaps  the  Government  customer  is  more  difficult  to  work  for  than  the  Prime  Con¬ 
tractor  customer.  This  survey  did  not  collect  sufficient  evidence  to  address  these  hypotheses. 

In  question  ProdOl,  acquirers  were  categorized  as  either  Army,  Navy,  Air  Force,  NASA,  DHS, 
DARPA,  Other  Government,  Commercial,  or  Other,  as  noted  in  Section  5.1.2.  Project  Perform¬ 
ance,  as  defined  in  Section  5. 1.5.4,  was  evaluated  in  each  category,  as  shown  in  Figure  37. 
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Figure  37:  Relationship  Between  Acquiring  Organization  and  Performance  ( Perf  Versus  Prod02) 

There  are  very  few  projects  in  each  category,  so  little  can  be  said  here  with  confidence.  Projects 
that  classified  themselves  in  the  Commercial  category  seem  to  have  slightly  better  success  than 
most  of  the  others.  At  least  based  on  this  data,  the  Navy  may  lack  a  middle  ground,  with  the  ma¬ 
jority  of  their  projects  delivering  either  Best  Performance  or  Lower  Performance,  but  very  few 
delivering  Moderate  performance.  Once  again,  care  should  be  taken  not  to  over  interpret  these 
differences. 

In  question  Prod04,  the  resulting  product’s  location  within  a  system’s  hierarchy  was  categorized 
as  either  System-of-Systems,  System,  Subsystem,  Component,  Process,  Material,  or  Other,  as 
noted  in  Section  5.1.2. 
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Figure  38:  Relationship  Between  Position  in  Systems  Hierarchy  and  Performance  ( Perf  Versus 
Prod04) 

As  shown  in  Figure  38,  when  compared  with  the  Project  Performance  composite  measure  defined 
in  Section  5. 1.5.4,  this  survey  question  shows  that  projects  delivering  Systems-of- Systems  are 
least  likely  (9%)  to  show  Best  Performance.  Of  the  projects  that  deliver  Systems,  38%  show  Best 
Performance.  Of  the  projects  that  deliver  Subsystems,  the  percentage  showing  Best  Performance 
drops  to  17%.  For  all  of  the  above  (SoS,  Systems,  and  Subsystems)  about  one  in  three  of  the  pro¬ 
jects  shows  lower  Performance.  The  numbers  of  projects  supplying  Components,  Processes,  or 
Materials  are  too  small  to  provide  meaningful  insight,  and  too  small  to  honor  our  promise  of  non¬ 
disclosure  made  to  the  survey  participants. 

In  question  Cont08,  the  project’s  utilization  of  Subcontractors  was  identified,  as  noted  in  Section 
5.1.2.  Its  relationship  with  the  Project  Performance  composite  measure  defined  in  Section  5. 1.5. 4, 
is  shown  in  Figure  39. 
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Figure  39:  Relationship  Between  Subcontracted  Percentage  and  Performance  (Perf  Versus  Cont08) 

The  percentage  of  subcontracted  effort  varied  across  the  project  sample  with  10  projects  subcon¬ 
tracting  less  than  5%  of  the  effort,  10  projects  subcontracting  5%  to  25%  of  the  effort,  17  projects 
subcontracting  25%  to  50%  of  the  effort  and  9  projects  subcontracting  more  than  50%  of  the  ef¬ 
fort.  No  clear  trend  is  apparent  among  the  projects  subcontracting  less  than  50%  of  the  effort,  re¬ 
gardless  of  the  amount  of  subcontracting.  However,  projects  exhibiting  Lower  Performance  in¬ 
creases  markedly  to  56%  among  those  projects  subcontracting  more  than  50%  of  their  project 
effort.  Likewise,  the  portion  exhibiting  Best  Performance  decreases  to  only  1 1%.  Possible  inter¬ 
pretations  of  this  observation  include 

•  Larger  subcontracting  efforts  require  more  coordination  among  subcontractors,  increasing  the 
difficulty  of  project  execution. 

•  Subcontracting  larger  amounts  of  the  project  decreases  the  control  that  can  be  exercised  on 
the  project. 

In  question  Contl4a,  the  project’s  Systems  Engineering  content  was  evaluated,  as  noted  in  Sec¬ 
tion  5.1.2.  This  parameter  showed  a  strong  negative  relationship  with  the  Project  Performance 
composite  measure  defined  in  Section  5. 1.5. 4,  as  shown  in  Figure  40. 
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Figure  40:  Relationship  Between  Systems  Engineering  Content  and  Performance  ( Perf  Versus 
Cont14a) 

However,  this  negative  relationship  is  somewhat  misleading.  Analysis  of  the  responses  to  ques¬ 
tion  Contl4a  reveals  that  the  projects  with  SE  content  greater  than  25%  seem  to  be  a  different 
type  of  project  (see  APPENDIX  D,  Section  D.l).  Excluding  these  projects  eliminates  the  negative 
relationship.  In  interpreting  Figure  40,  we  will  concentrate  primarily  on  the  projects  with  SE  con¬ 
tent  <  25%. 

Planned  Systems  Engineering  effort  varied  across  the  project  sample  with  three  projects  having 
SE  Content  <  5%,  10  having  SE  Content  from  5  to  10%,  nine  having  SE  Content  from  10  to  15%, 
and  eight  having  SE  Content  from  1 5  to  25%.  Regardless  of  the  amount  of  SE  content,  approxi¬ 
mately  one-third  of  the  projects  exhibited  Lower  Performance.  The  largest  percentage  of  projects 
achieving  the  Best  Performance  seemed  to  occur  for  SE  Content  levels  from  10  to  15%.  However, 
care  should  be  exercised  in  interpreting  the  meaning  of  the  difference  based  on  such  a  small  num¬ 
ber  of  cases  in  each  of  the  categories  of  SE  Content. 

5.2.2. 1  CMMI-Based  Process  Management  Versus  Performance  (  SEC  cmmi  vs.  Perf) 

The  CMMI-related  supplier  capability  composite  measure  described  in  Section  5. 1.2.1  showed 
weak  positive  relationship  with  the  Project  Perfonnance  composite  measure  defined  in  Section 
5. 1.5.4,  as  shown  in  Figure  41. 
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Figure  41:  Relationship  Between  Supplier  CMMI-Related  Capability  and  Project  Performance  (Perf 
Versus  SECcmmi) 

CMMI-related  capability  varied  across  the  project  sample  with  14  projects  exhibiting  lower 
CMMI-related  capability,  14  exhibiting  moderate  CMMI-related  capability,  and  18  exhibiting 
higher  CMMI-related  capability.  A  weak  positive  relationship  between  SECcmmi  and  Perf  is  evi¬ 
dent.  Among  projects  exhibiting  the  Best  Performance,  7%  of  the  projects  with  lower  CMMI- 
related  capability  exhibited  Best  Performance,  while  39%  of  projects  with  higher  CMMI-related 
capability  exhibited  Best  Performance.  However,  there  are  only  small  differences  in  Lower  Per¬ 
formance  as  a  function  of  Supplier  CMMI-related  capability. 

A  Gamma  value  of  0.22  indicates  that  there  is  a  weak  positive  relationship  between  Project  Per¬ 
formance  and  the  elements  of  CMMI-related  capabilities  addressed  in  this  survey.  A  p  value  of 
0.13  indicates  that  this  interpretation  is  not  reliable  since  there  exists  a  13%  probability  that  a  rela¬ 
tionship  of  this  magnitude  could  spontaneously  occur  by  chance  alone. 

There  are  many  possible  reasons  for  this  relatively  weak  statistical  relationship.  In  particular,  two 
of  the  three  questions  that  are  combined  in  the  measure  of  Supplier  CMMI-related  capability  ask 
about  organizational  maturity  level,  rather  than  about  the  Systems  Engineering  management  and 
engineering  capabilities  of  the  projects  where  the  development  work  is  done.6  As  noted  through¬ 
out  Section  5.2.3,  the  statistical  relationships  with  Project  Performance  are  considerably  stronger 
for  most  of  the  measures  of  supplier  Systems  Engineering  Capability. 


6  The  full  nature  of  the  relationship  between  organizational  maturity  and  project  capabilities  needs  to  be  analyzed 
more  fully  elsewhere.  The  data  from  this  survey  are  only  tangential  to  a  fuller  analysis;  however,  they  do  appear 
to  show  some  significant  differences  in  project  SE  Capability  as  a  function  of  organizational  maturity  level. 


66  |  CMU/SEI-2007-SR-014 


ND1A 


5. 2. 2. 2  Process  Improvement  vs.  Performance  (SECimp  versus  Perf ) 


The  Process  Improvement  supplier  capability  measure  is  described  in  Section  5. 1.2. 3.  Its  relation¬ 
ship  with  the  Project  Performance  composite  measure  defined  in  Section  5. 1.5.4  is  shown  in 
Figure  42. 
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Figure  42:  Relationship  between  Process  Improvement  capability  and  Project  Performance  (Perf  Ver¬ 
sus  SECimp) 

Process  Improvement  capability  varied  across  the  project  sample  with  20  projects  exhibiting  low¬ 
er  Process  Improvement  capability,  14  exhibiting  moderate  Process  Improvement  capability,  and 
12  exhibiting  higher  Process  Improvement  capability.  Overall,  a  very  weak  positive  relationship 
between  SECimp  and  Perf  is  evident.  Among  projects  exhibiting  the  Best  Performance,  20%  of 
the  projects  with  lower  Process  Improvement  capability  exhibited  Best  Performance,  while  42% 
of  projects  with  higher  Process  Improvement  capability  exhibited  Best  Performance.  However, 
the  relationship  is  less  consistent  with  respect  to  Lower  Perfonnance,  where  the  projects  with  the 
least  Process  Improvement  capability  in  fact  fared  somewhat  better  than  did  those  that  exhibited 
moderate  or  higher  capability. 

A  Gamma  value  of  0.05  indicates  that  there  is  a  weak  to  non-existent  positive  overall  relationship 
between  Project  Performance  and  the  elements  of  Process  Improvement  addressed  in  this  survey. 
A  p  value  of  0.39  indicates  that  there  is  a  39%  probability  that  a  relationship  of  this  magnitude 
could  spontaneously  occur  by  chance  alone. 

5.2.2. 3  Prior  Experience  vs.  Performance  (  SEC  exp  vs.  Perf) 

The  Prior  Experience  composite  measure  is  described  in  Section  5. 1.2.2.  Its  relationship  with  the 
Project  Performance  composite  measure  defined  in  Section  5. 1.5. 4  is  shown  in  Figure  43. 
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Figure  43:  Relationship  between  Prior  Experience  and  Project  Performance  (Perf  Versus  SEC  exp) 

Prior  Experience  varied  across  the  project  sample  with  14  projects  exhibiting  lower  Prior  Experi¬ 
ence,  18  exhibiting  moderate  Prior  Experience,  and  14  exhibiting  higher  Prior  Experience.  Those 
projects  with  more  experience  were  somewhat  more  likely  to  demonstrate  Best  Performance; 
however,  the  other  differences  are  not  consistent. 


5.2.3  Relationships  between  Systems  Engineering  Capabilities  (SEC)  and  Perform¬ 
ance  (Perf) 


To  identify  the  relationship  between  Systems  Engineering  Capabilities  and 


we  will  examine  the  following  relationships: 

•  IPT-based  capability  versus  Performance  ( 

•  Project  Planning  versus  Performance  ( 

•  Project  Monitoring  and  Control  versus  Perfonnance  ( 

•  Risk  Management  versus  Performance  ( 

•  Requirements  Development  and  Management  versus  Performance  ( 

•  Trade  Studies  versus  Performance  ( 

•  Product  Architecture  versus  Performance  ( 

•  Technical  Solution  (=  SECjrade  +  SECarch)  versus  Performance  ( 

•  Product  Integration  versus  Performance  ( 

•  Verification  versus  Performance  ( 

•  Validation  versus  Performance  ( 

•  Configuration  Management  versus  Performance  ( 

•  Total  Systems  Engineering  Capability  versus  Performance  ( 

•  Req’ts  and  Technical  Solution  Capability  versus  Performance  ( 
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5.2. 3.1  IPT-related  capability  versus  Performance  (  SEC/pr  versus  Perf ) 


The  Integrated  Product  Team  (IPT)  supplier  capability  composite  measure  described  in  Section 
5. 1.3.1  showed  a  moderately  strong  positive  relationship  with  the  Project  Performance  composite 
measure  defined  in  Section  5. 1.5. 4,  as  shown  in  Figure  44. 
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Figure  44:  Relationship  Between  Supplier  IPT  Capability  and  Project  Performance  (Perf  Versus 
SECipt) 

Utilization  of  IPTs  varied  across  the  project  sample  with  15  projects  exhibiting  low  IPT  utiliza¬ 
tion,  1 6  exhibiting  moderate  IPT  utilization,  and  1 5  exhibiting  high  IPT  utilization.  A  moderately 
strong  positive  relationship  between  SECipt  and  Perf  is  evident.  Among  projects  exhibiting  the 
Best  Performance,  only  13%  of  projects  with  low  IPT  utilization  exhibited  Best  Performance, 
while  53%  of  projects  with  high  IPT  utilization  exhibited  Best  Performance.  Similarly,  only  20% 
of  projects  with  High  IPT  utilization  exhibited  Lower  Performance.  However,  the  differences  be¬ 
tween  lower  and  moderate  performance  are  less  consistent  among  the  projects  that  exhibit  lower 
as  compared  to  moderate  Supplier  IPT  capability. 

A  Gamma  value  of  0.34  indicates  that  there  is  a  moderately  strong  positive  relationship  between 
Project  Performance  and  the  elements  of  Integrated  Project  Team  deployment  addressed  in  this 
survey.  A  p  value  of  0.04  indicates  that  there  is  a  4%  probability  that  a  relationship  of  this  magni¬ 
tude  could  spontaneously  occur  by  chance  alone. 

Among  Best  Performing  Projects,  the  percentage  of  projects  exhibiting  high  IPT  process  capabil¬ 
ity  (53%)  is  among  the  highest  across  the  process  areas  analyzed.  This  may  indicate  the  value  of 
integrated  teams  and  collaboration  in  helping  to  produce  successful  project  outcomes. 
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5.2. 3.2  Project  Planning  versus  Performance  (  SECpp  versus  Perf) 


The  supplier’s  Project  Planning  capability  composite  measure  described  in  Section  5. 1.3. 2 
showed  a  weak  positive  relationship  with  the  Project  Performance  composite  measure  defined  in 
Section  5. 1.5.4,  as  shown  in  Figure  45. 
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Figure  45:  Relationship  between  Supplier  Project  Planning  Capability  and  Project  Performance  (Perf 
Versus  SECpp) 

Project  Planning  capability  varied  across  the  project  sample  with  13  projects  exhibiting  lower  Pro¬ 
ject  Planning  capability,  14  exhibiting  moderate  Project  Planning  capability,  and  19  exhibiting 
higher  Project  Planning  capability.  A  weak  positive  relationship  between  SECPP  and  Perf  is  evi¬ 
dent.  Among  the  projects  exhibiting  Best  Performance,  only  15%  of  the  projects  with  lower  Pro¬ 
ject  Planning  capability  exhibited  Best  Performance,  while  37%  of  projects  with  the  higher  Pro¬ 
ject  Planning  capability  exhibited  Best  Performance.  Flowever,  the  higher  capability  projects  were 
as  likely  to  exhibit  lower  performance  as  high.  This  inconsistency  may  be  due  to  misinterpretation 
by  the  responding  projects  of  the  intended  meaning  of  the  survey  questions  or  to  other  measure¬ 
ment  error.  Moreover,  regardless  of  the  reasons,  no  single  measure  can  be  expected  to  account  for 
all  of  the  variation  in  Project  Perfonnance. 

A  Gamma  value  of  0.13  indicates  that  there  is  a  weak  positive  relationship  between  Project  Per¬ 
formance  and  the  elements  of  Project  Planning  addressed  in  this  survey.  A  p  value  of  0.24  indi¬ 
cates  that  there  is  a  24%  probability  that  a  relationship  of  this  magnitude  could  spontaneously  oc¬ 
cur  by  chance  alone. 

5.2. 3. 3  Project  Monitoring  and  Control  versus  Performance  (  SEC PMC  versus  Perf ) 

The  Project  Monitoring  and  Control  supplier  capability  composite  measure  described  in  Section 

5. 1.3.3  showed  weak  negative  relationship  with  the  Project  Performance  composite  measure  de¬ 
fined  in  Section  5. 1.5.4,  as  shown  in  Figure  46. 
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Figure  46:  Relationship  Between  Supplier  Project  Monitoring  and  Control  Capability  and  Project  Per¬ 
formance  (Perf  Versus  SECpmc) 

Project  Monitoring  and  Control  capability  varied  across  the  project  sample  with  13  projects  exhib¬ 
iting  lower  Project  Monitoring  and  Control  capability,  13  exhibiting  moderate  Project  Monitoring 
and  Control  capability,  and  20  exhibiting  higher  Project  Monitoring  and  Control  capability.  A 
weak  negative  relationship  between  SECpmc  and  Perf  is  evident.  Among  projects  exhibiting  the 
Best  Performance,  a  positive  influence  of  SECPMc  was  seen,  with  23%  of  the  projects  with  the 
lower  Project  Monitoring  and  Control  capability  exhibiting  Best  Performance,  while  30%  of  pro¬ 
jects  with  higher  Project  Monitoring  and  Control  capability  exhibited  Best  Performance.  The  per¬ 
centage  of  poorly  performing  projects  showed  negative  influences  of  SECpmc  on  Project  Per¬ 
formance.  Twenty-three  percent  of  the  projects  with  lower  Project  Monitoring  and  Control 
capability  exhibited  lower  Performance,  while  45%  of  projects  with  higher  Project  Monitoring 
and  Control  capability  exhibited  lower  Performance. 

A  Gamma  value  of  -0.13  indicates  that  there  is  a  weak  negative  relationship  between  Project  Per¬ 
formance  and  the  elements  of  Project  Monitoring  and  Control  addressed  in  this  survey.  A  p  value 
of  0.25  indicates  that  there  is  a  25%  probability  a  relationship  of  this  magnitude  could  spontane¬ 
ously  occur  by  chance  alone. 

This  finding  raises  the  question,  “How  could  more  monitoring  and  control  degrade  Project  Per¬ 
formance?”  But  perhaps  that  is  the  wrong  question  to  ask.  Remember  that  the  analysis  shows  that 
there  is  a  relationship  between  Performance  and  Project  Monitoring  and  Control;  it  does  not  re¬ 
veal  which  is  the  cause  and  which  is  the  effect.  When  a  project  encounters  difficulties,  often  in¬ 
creased  scrutiny  is  one  of  the  first  actions  taken.  Thus,  rather  than  interpreting  the  results  as 

“Increased  Project  Monitoring  and  Control  results  in  Reduced  Project  Performance” 

The  more  correct  interpretation  may  be 


NATIONAL  DEFENSE  INDUSTRIAL  ASSOCIATION 


SOFTWARE  ENGINEERING  INSTITUTE  |  71 


“Poorly  Performing  projects  result  in  Increased  Project  Monitoring  and  Control.” 

5.2. 3.4  Risk  Management  versus  Performance  (  SECrskm  versus  Perf) 

The  Risk  Management  supplier  capability  composite  measure  described  in  Section  5. 1.3. 4  showed 
a  moderately  strong  positive  relationship  with  the  Project  Performance  composite  measure  de¬ 
fined  in  Section  5. 1.5. 4,  as  shown  in  Figure  47. 
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Figure  47:  Relationship  Between  Supplier  Risk  Management  Capability  and  Project  Performance  (Perf 
Versus  SECrskm) 

Risk  Management  capability  varied  across  the  project  sample  with  17  projects  exhibiting  lower 
Risk  Management  capability,  15  exhibiting  moderate  Risk  Management  capability,  and  14  exhib¬ 
iting  higher  Risk  Management  capability.  A  moderately  strong  positive  relationship  between 
SECrskm  and  Perf  is  evident.  Among  projects  exhibiting  the  Best  Performance,  only  18%  of  the 
projects  with  lower  Risk  Management  capability  exhibited  Best  Performance,  while  64%  of  pro¬ 
jects  with  higher  Risk  Management  capability  exhibited  Best  Performance.  The  percentage  of 
poorly  performing  projects  remained  largely  unchanged  as  a  function  of  Risk  Management  capa¬ 
bility.  However,  among  the  projects  with  lower  or  moderate  Risk  Management  capabilities,  a  sig¬ 
nificant  number  exhibited  only  moderate  Project  Performance. 

A  Gamma  value  of  0.28  indicates  that  there  is  a  moderately  strong  positive  relationship  between 
Project  Performance  and  the  elements  of  Risk  Management  addressed  in  this  survey.  A p  value  of 
0.06  indicates  that  there  is  a  6%  probability  a  relationship  of  this  magnitude  could  spontaneously 
occur  by  chance  alone. 

Nearly  2/3  of  the  Best  Performing  projects  exhibited  a  higher  capability  in  Risk  Management.  In 
fact,  among  Best  Performing  projects,  the  composite  score  for  Risk  Management  capability  (64%) 
was  the  highest  among  all  analyzed  process  areas  by  a  significant  margin.  This  may  suggest  the 
value  of  effective  Risk  Management  in  producing  successful  project  outcomes. 
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5.2. 3. 5  Requirements  Development  versus  Performance  (  SEC req  versus  Perf) 

The  Requirements  Development  and  Management  supplier  capability  composite  measure  de¬ 
scribed  in  Section  5. 1.3. 5  showed  a  moderately  strong  positive  relationship  with  the  Project  Per¬ 
formance  measure  defined  in  Section  5. 1.5. 4,  as  shown  in  Figure  48. 
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Figure  48:  Relationship  Between  Supplier  Requirements  Capabilities  and  Project  Performance  (Perf 
Versus  SECreq) 

Requirements  capability  varied  across  the  project  sample  with  16  projects  exhibiting  lower  Re¬ 
quirements  capability,  19  exhibiting  moderate  Requirements  capability,  and  1 1  exhibiting  higher 
Requirements  capability.  A  moderately  strong  positive  relationship  between  SECreq  and  Perf  is 
evident.  Among  projects  exhibiting  the  Best  Perfonnance,  only  18%  of  the  projects  with  lower 
Requirements  capability  exhibited  Best  Performance,  while  55%  of  projects  with  higher  Re¬ 
quirements  capability  exhibited  Best  Performance.  Similarly,  44%  of  the  projects  with  the  lower 
Requirements  capability  exhibited  lower  Performance,  while  only  27%  of  projects  with  the  higher 
Requirements  capability  exhibited  lower  Performance. 

A  Gamma  value  of  0.33  indicates  that  there  is  a  moderately  strong  positive  relationship  between 
Project  Performance  and  the  elements  of  Requirements  Development  and  Management  addressed 
in  this  survey.  A  p  value  of  0.04  indicates  that  there  is  a  4%  probability  that  a  relationship  of  this 
magnitude  could  spontaneously  occur  by  chance  alone. 

Over  half  of  the  fiigher  Performing  Projects  exhibited  a  higher  capability  in  Requirements  Devel¬ 
opment  and  Management,  suggesting  the  value  of  effective  requirements  management  in  produc¬ 
ing  successful  project  outcomes 
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5.2. 3. 6  Trade  Studies  versus  Performance  (  SEC  trade  versus  Perf) 


The  Trade  Studies  supplier  capability  composite  measure  described  in  Section  5. 1.3. 6  showed  a 
moderately  strong  to  strong  positive  relationship  with  the  Project  Performance  composite  measure 
defined  in  Section  5. 1.5.4,  as  shown  in  Figure  49. 
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Figure  49:  Relationship  Between  Supplier  Trade  Study  Capabilities  and  Project  Performance  (Perf 
Versus  SECtrade) 

Trade  Study  capability  varied  across  the  project  sample  with  18  projects  exhibiting  lower  Trade 
Study  capability,  12  exhibiting  moderate  Trade  Study  capability,  and  16  exhibiting  higher  Trade 
Study  capability.  A  moderately  strong  to  strong  positive  relationship  between  SECTK4DE  and  Perf 
is  evident.  Among  projects  exhibiting  the  Best  Performance,  only  17%  of  the  projects  with  lower 
Trade  Study  capability  exhibited  Best  Performance,  while  50%  of  projects  with  higher  Trade 
Study  capability  exhibited  Best  Perfonnance.  Similarly,  39%  of  the  projects  with  lower  Trade 
Study  capability  exhibited  lower  Performance,  while  only  19%  of  projects  with  higher  Trade 
Study  capability  exhibited  lower  Performance. 

A  Gamma  value  of  0.37  indicates  that  there  is  a  moderately  strong  to  strong  positive  relationship 
between  Project  Performance  and  the  elements  of  Trade  Study  Capabilities  addressed  in  this  sur¬ 
vey.  A  p  value  of  0.03  indicates  that  there  is  a  3%  probability  that  a  relationship  of  this  magnitude 
could  spontaneously  occur  by  chance  alone. 

5.2. 3.7  Product  Architecture  versus  Performance  (  SEC  arch  versus  Perf) 

The  Product  Architecture  supplier  capability  composite  measure  described  in  Section  5. 1.3. 7 
showed  a  moderately  strong  to  strong  positive  relationship  with  the  Project  Performance  compos¬ 
ite  measure  defined  in  Section  5. 1.5. 4,  as  shown  in  Figure  50. 
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Figure  50:  Relationship  Between  Supplier  Product  Architecture  Capabilities  and  Project  Performance 
(Perf  Versus  SECarch) 

Product  Architecture  capability  varied  across  the  project  sample  with  18  projects  exhibiting  lower 
Product  Architecture  capability,  14  exhibiting  moderate  Product  Architecture  capability,  and  13 
exhibiting  higher  Product  Architecture  capability.  A  moderately  strong  to  strong  positive  relation¬ 
ship  between  SECarch  and  Perf  is  evident.  Among  projects  exhibiting  the  Best  Performance,  on¬ 
ly  11%  of  the  projects  with  the  lower  Product  Architecture  capability  exhibited  Best  Performance, 
while  46%  of  projects  with  the  higher  Product  Architecture  capability  exhibited  Best  Perform¬ 
ance.  Similarly,  44%  of  the  projects  with  the  lower  Product  Architecture  capability  exhibited 
lower  Performance,  while  only  23%  of  projects  with  the  higher  Product  Architecture  capability 
exhibited  lower  Performance. 

A  Gamma  value  of  0.40  indicates  that  there  is  a  moderately  strong  to  strong  positive  relationship 
between  Project  Performance  and  the  elements  of  Product  Architecture  addressed  in  this  survey. 

A  p  value  of  0.02  indicates  that  there  is  a  2%  probability  that  a  relationship  of  this  magnitude 
could  spontaneously  occur  by  chance  alone. 

This  data  seems  to  substantiate  the  widely-held  belief  in  the  importance  of  effective  architectural 
practices  in  producing  successful  project  outcomes. 

5.2. 3. 8  Technical  Solution  (=  SECtrade  +  SECarch)  versus  Performance  (  SECrs  versus  Perf) 

The  Technical  Solution  supplier  capability  composite  measure  described  in  Section  5. 1.3. 8 
showed  a  moderately  strong  positive  relationship  with  the  Project  Performance  composite  meas¬ 
ure  defined  in  Section  5. 1.5.4,  as  shown  in  Figure  51. 
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Figure  51:  Relationship  Between  Supplier  Technical  Solution  Capabilities  and  Project  Performance 
(Perf  Versus  SECts) 

Technical  Solution  capability  varied  across  the  project  sample  with  15  projects  exhibiting  lower 
Technical  Solution  capability,  15  exhibiting  moderate  Technical  Solution  capability,  and  15  ex¬ 
hibiting  higher  Technical  Solution  capability.  A  moderately  strong  positive  relationship  between 
SECts  and  Perf  is  evident.  Among  projects  exhibiting  the  Best  Performance,  only  7%  of  the  pro¬ 
jects  with  the  lower  Technical  Solution  capability  exhibited  Best  Performance,  while  46%  of  pro¬ 
jects  with  the  higher  Technical  Solution  capability  exhibited  Best  Performance.  Consistent,  but 
smaller  differences  were  seen  among  the  lower  Performance  projects.  Forty  percent  of  the  projects 
with  lower  Technical  Solution  capability  exhibited  lower  Performance,  while  only  27%  of  pro¬ 
jects  with  higher  Technical  Solution  capability  exhibited  lower  Performance. 

A  Gamma  value  of  0.36  indicates  that  there  is  a  moderately  strong  positive  relationship  between 
Project  Performance  and  the  elements  of  Product  Architecture  addressed  in  this  survey.  A p  value 
of  0.03  indicates  that  there  is  a  3%  probability  a  relationship  of  this  magnitude  could  spontane¬ 
ously  occur  by  chance  alone. 

5.2. 3.9  Product  Integration  versus  Performance  (  SECP/  versus  Perf) 

The  Product  Integration  supplier  capability  measure  described  in  Section  5. 1.3. 9  showed  a  weak 
positive  relationship  with  the  Project  Performance  composite  measure  defined  in  Section  5. 1.5. 4, 
as  shown  in  Figure  52. 


76  |  CMU/SEI-2007-SR-014 


NDIA 


Lower 

Capability 

(x=  1) 

N  =  14 


Moderate 
Capability 
(x  =  2  or  3) 
N  =  24 


Higher 
Capability 
(x  =  4) 

N  =  7 


Gamma  =  0.21 

p  =  0.16 


Figure  52:  Relationship  Between  Supplier  Product  Integration  and  Project  Performance  (Perf  Versus 
SECpi) 


Product  Integration  capability  varied  across  the  project  sample  with  14  projects  exhibiting  lower 
Product  Integration  capability,  24  exhibiting  moderate  Product  Integration  capability,  and  7  exhib¬ 
iting  higher  Product  Integration  capability.  A  weak  positive  relationship  between  SECpj  and  Perf 
is  evident.  Among  projects  exhibiting  the  Best  Perfonnance,  only  14%  of  the  projects  with  the 
lower  Product  Integration  capability  exhibited  Best  Performance,  while  43%  of  projects  with  the 
higher  Product  Integration  capability  exhibited  Best  Performance.  Consistent,  but  much  smaller 
differences  were  seen  among  the  lower  Performance  projects.  Thirty-six  percent  of  the  projects 
with  lower  Product  Integration  capability  exhibited  lower  Performance,  while  only  29%  of  pro¬ 
jects  with  higher  Product  Integration  capability  exhibited  lower  Perfonnance. 

Note,  however,  that  the  weak  relationship  in  this  instance  may  be  due  to  the  fact  that  the  Product 
Integration  capability  measure  is  based  on  only  a  single  survey  question.  Moreover,  the  answers 
to  that  question  are  not  evenly  distributed  across  the  possible  response  categories  from  “disagree 
strongly”  to  “agree  strongly.”  It  is  quite  possible  that  a  composite  measure  of  Product  Integration 
capability  would  reduce  the  measurement  error  typically  found  in  a  single  survey  question.  Of 
course  we  cannot  know  from  this  survey,  but  the  statistical  relationship  with  Project  Performance 
then  might  well  be  comparable  to  those  found  with  many  of  the  other  similar  Systems  Engineer¬ 
ing  Capability  measures  in  this  survey. 


A  Gamma  value  of  0.21  indicates  that  there  is  a  weak  positive  relationship  between  Project  Per¬ 
formance  and  the  elements  of  Product  Architecture  addressed  in  this  survey.  A  p  value  of  0.16 
indicates  that  there  is  a  16%  probability  that  a  relationship  of  this  magnitude  could  spontaneously 
occur  by  chance  alone. 
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5.2.3.10  Verification  versus  Performance  (  SECi/er  versus  Perf ) 


The  Verification  supplier  capability  composite  measure  described  in  Section  5.1.3.10  showed  a 
moderately  strong  positive  relationship  with  the  Project  Performance  composite  measure  defined 
in  Section  5. 1.5.4,  as  shown  in  Figure  53. 
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Figure  53:  Relationship  Between  Supplier  Verification  Capabilities  and  Project  Performance  (Perf  Ver¬ 
sus  SECver) 

Verification  capability  varied  across  the  project  sample  with  16  projects  exhibiting  lower  Verifi¬ 
cation  capability,  15  exhibiting  moderate  Verification  capability,  and  15  exhibiting  higher  Verifi¬ 
cation  capability.  A  moderately  strong  positive  relationship  between  SECver  and  Perf  is  evident. 
Among  projects  exhibiting  the  Best  Perfonnance,  only  7%  of  the  projects  with  the  lower  Verifica¬ 
tion  capability  exhibited  Best  Performance,  while  47%  of  projects  with  the  higher  Verification 
capability  exhibited  Best  Performance.  The  percentage  of  poorly  performing  projects  remained 
largely  unchanged  as  a  function  of  Verification  capability. 

A  Gamma  value  of  0.25  indicates  that  there  is  a  moderately  strong  positive  relationship  between 
Project  Perfonnance  and  the  elements  of  Verification  addressed  in  this  survey.  A  p  value  of  0.09 
indicates  that  there  is  a  9.3%  probability  that  a  relationship  of  this  magnitude  could  spontaneously 
occur  by  chance  alone. 

5.2.3.11  Validation  versus  Performance  (  SECwu  versus  Perf ) 

The  Validation  supplier  capability  composite  measure  described  in  Section  5. 1.3.1 1  showed  a 
moderately  strong  positive  relationship  with  the  Project  Performance  composite  measure  defined 
in  Section  5. 1.5. 4,  as  shown  in  Figure  54. 
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Figure  54:  Relationship  Between  Supplier  Validation  Capabilities  and  Project  Performance  (Perf  Ver¬ 
sus  SECval) 

Validation  capability  varied  across  the  project  sample  with  13  projects  exhibiting  lower  Valida¬ 
tion  capability,  12  exhibiting  moderate  Validation  capability,  and  21  exhibiting  higher  Validation 
capability.  A  moderately  strong  positive  relationship  between  SECVEr  and  Perf  is  evident. 
Among  projects  exhibiting  the  Best  Performance,  23%  of  the  projects  with  the  lower  Validation 
capability  exhibited  Best  Performance,  while  38%  of  projects  with  the  higher  Validation  capabil¬ 
ity  exhibited  Best  Performance.  Similar  differences  were  seen  among  the  lower  Performance  pro¬ 
jects.  Fifty-four  percent  of  the  projects  with  lower  Validation  capability  exhibited  lower  Perform¬ 
ance,  while  only  29%  of  projects  with  higher  Validation  capability  exhibited  lower  Performance. 

A  Gamma  value  of  0.28  indicates  that  there  is  a  moderately  strong  positive  relationship  between 
Project  Performance  and  the  elements  of  Validation  addressed  in  this  survey.  A  p  value  of  0.07 
indicates  that  there  is  a  7%  probability  that  a  relationship  of  this  magnitude  could  spontaneously 
occur  by  chance  alone. 

It  is  interesting  to  note  that,  among  Lower  Performing  projects,  the  Validation  process  area  has 
the  greatest  percentage  of  Lower  Capability  scores  (54%),  by  a  significant  margin  over  all  other 
composite  process  area  scores  analyzed  in  this  survey.  In  other  words,  weaknesses  in  understand¬ 
ing  and  validating  user  needs  may  be  a  significant  factor  in  Project  Perfonnance  issues.  However, 
the  inverse  inference  (i.e.,  higher  Validation  process  capability  leading  to  higher  Project  Perform¬ 
ance)  cannot  be  supported  by  the  available  data. 

5.2.3.12  Configuration  Management  versus  Performance  (  SEC  cm  versus  Perf) 

The  Configuration  Management  supplier  capability  composite  measure  described  in  Section 

5.1.3.12  showed  a  weak  positive  relationship  with  the  Project  Performance  composite  measure 
defined  in  Section  5. 1.5.4,  as  shown  in  Figure  55. 
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Figure  55:  Relationship  Between  Supplier  Configuration  Management  Capabilities  and  Project  Per¬ 
formance  (Perf  Versus  SECcm) 

Configuration  Management  capability  varied  across  the  project  sample  with  17  projects  exhibiting 
lower  Configuration  Management  capability,  1 1  exhibiting  moderate  Configuration  Management 
capability,  and  1 8  exhibiting  higher  Configuration  Management  capability.  A  weak  positive  rela¬ 
tionship  between  SECcm  and  Perf  is  evident.  Among  projects  exhibiting  the  Best  Performance, 
24%  of  the  projects  with  the  lower  Configuration  Management  capability  exhibited  Best  Perform¬ 
ance,  while  39%  of  projects  with  the  higher  Configuration  Management  capability  exhibited  Best 
Performance.  Among  the  lower  Performance  projects,  less  consistent  differences  are  seen  else¬ 
where  in  Figure  55. 

A  Gamma  value  of  0.13  indicates  that  there  is  a  weak  positive  relationship  between  Project  Per¬ 
formance  and  the  elements  of  Configuration  Management  addressed  in  this  survey.  A  p  value  of 
0.26  indicates  that  there  is  a  26%  probability  that  a  relationship  of  this  magnitude  could  spontane¬ 
ously  occur  by  chance  alone. 


Note  that  the  threshold  for  Lower  Capability  (3.0)  is  the  highest  in  absolute  terms  among  all  com¬ 
posite  capability  scores  analyzed.  This,  coupled  with  relatively  even  distribution  across  Project 
Performance  categories  and  a  weak  positive  Gamma  relationship  (0.13)  with  high  p  value,  may 
indicate  that  Configuration  Management  practices  are  well  accepted  and  implemented  across  pro¬ 
jects  (i.e.,  common-place  and  not  a  significant  discriminator  in  Project  Performance).  This  inter¬ 
pretation  is  supported  by  a  high  percentage  of  ‘Strongly  Agree’  CM  practices  as  depicted  in  Sec¬ 
tion  D.3.12. 


5.2.3.13  Overall  Systems  Engineering  Capability  versus  Performance  (  SEC  versus  Perf) 

The  Overall  Systems  Engineering  Capability  ( SEC)  composite  measure  described  in  Section 

5.1.3.13  showed  a  moderately  strong  positive  statistical  relationship  with  the  Project  Perfonnance 
measure  defined  in  Section  5. 1.5.4,  as  shown  in  Figure  56. 


80  |  CMU/SEI-2007-SR-014 


NDIA 


Lower 
Capability 
(x  <2. 5) 
N  =  13 


Moderate 
Capability 
(2.5<x<3.0) 
N  =  17 


Higher 
Capability 
(x  >3.0) 
N  =  16 


Gamma  =  0.32 
p  =  0.04 


Figure  56:  Relationship  Between  Overall  Systems  Engineering  Capability  and  Project  Performance 
(Perf  Versus  SEC) 

Overall  Systems  Engineering  Capability  varied  across  the  project  sample  with  13  projects  exhibit¬ 
ing  Lower  Overall  SE  Capability,  17  exhibiting  Moderate  Overall  SE  Capability,  and  16  exhibit¬ 
ing  Eligher  Overall  SE  Capability.  A  moderately  strong  positive  relationship  between  SEC  and 
Perf  is  evident.  Among  projects  exhibiting  the  Best  Performance,  only  15%  of  projects  with  lower 
Overall  SE  Capability  exhibited  Best  Perfonnance,  while  56%  of  projects  with  higher  Overall  SE 
Capability  exhibited  Best  Performance.  The  other  differences  in  Figure  56  are  less  consistent  and 
less  pronounced. 

The  relationship  between  Overall  Systems  Engineering  Capability  and  Project  Performance  suc¬ 
cinctly  summarizes  the  overall  effect  of  the  Systems  Engineering  practices  covered  in  this  survey. 
However,  the  relationship  remains  only  moderately  strong  since  the  capability  measure  is  based 
on  all  of  the  practices  including  those  that  appear  to  have  a  less  direct  effect  on  Project  Perform¬ 
ance. 

A  Gamma  value  of  0.32  indicates  that  there  is  a  moderately  strong  positive  relationship  between 
Project  Performance  and  the  elements  of  Overall  SE  Capability  addressed  in  this  survey.  A p  val¬ 
ue  of  0.04  indicates  that  there  is  a  4%  probability  that  a  relationship  of  this  magnitude  could  spon¬ 
taneously  occur  by  chance  alone. 

5.2.3.14  Combined  Requirements  and  Technical  Solution  Capability  versus  Performance 
(SECr+ts  versus  Perf ) 

The  moderately  strong  statistical  relationships  between  Systems  Engineering  Capabilities  and 
Project  Performance  just  described  are  notable  by  themselves.  However,  selectively  combining 
more  than  one  of  the  composite  capability  measures  that  are  most  strongly  related  to  Project  Per- 
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formance  can  yield  a  notably  stronger  relationship.  For  example,  we  created  a  higher  order  SE 
Capability  measure  by  combining  the  Requirements  ( SECreq )  and  Technical  Solution  ( SECTs ) 
composite  measures  together  into  a  single  composite  measure  ( SECr+ts )  This  was  done  simply  by 
equally  weighting  the  contribution  of  the  original  two  composite  scores. 

Because  of  the  small  number  of  projects,  we  are  unable  to  do  rigorous  multivariate  statistical  ana¬ 
lyses  of  the  combined  effect  of  several  measures  on  Project  Performance  ( Perf ).  Instead  we  have 
created  composite  capability  measures  based  on  the  statistical  relationships  between  two  or  more 
other  measures  that  themselves  are  related  to  Project  Performance  (Perf).  As  we  have  done 
throughout  the  report,  the  new  combined  measures  are  separated  into  three  categories  since  a  rela¬ 
tively  small  number  of  projects  participated  in  the  survey. 


The  Combined  Requirements  and  Technical  Solution  Capability  composite  measure 
( S EC n + /.sj s h o we d  a  strong  positive  statistical  relationship  with  the  Project  Performance  measure 
defined  in  Section  5. 1.5.4,  as  shown  in  Figure  57 
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Figure  57:  Relationship  between  Combined  Requirements  and  Technical  Solution  Capability 
and  Project  Performance  (Perf  Versus  SECr+ts) 

Systems  Engineering  Capability  varied  across  the  survey  sample  with  15  projects  exhibiting  Low¬ 
er  Combined  Requirements  and  Technical  Solution  Capability,  13  exhibiting  Moderate  Combined 
Requirements  and  Technical  Solution  Capability,  and  18  exhibiting  Higher  Combined  Require¬ 
ments  and  Technical  Solution  Capability.  Among  projects  exhibiting  the  Best  Performance,  only 
13%  of  projects  with  Low  capability  exhibited  Best  Performance,  while  50%  of  projects  with 
High  capability  exhibited  Best  Performance.  Lower  Performance  projects  also  showed  similar 
differences,  with  53%  of  projects  with  Lower  Combined  Requirements  and  Technical  Solution 
Capability  exhibiting  Lower  Performance,  while  only  22%  of  projects  with  Higher  Combined 
Requirements  and  Technical  Solution  Capability  exhibited  Lower  Performance. 
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A  Gamma  value  of  0.49  indicates  that  there  is  a  strong  relationship  between  Project  Performance 
and  the  elements  of  Requirements  and  Technical  Solution  deployment  addressed  in  this  survey.  A 
p  value  of  0.005  indicates  that  there  is  a  0.5%  probability  that  this  type  of  relationship  could  occur 
by  chance  alone. 

5.2.4  Relationships  between  Acquirer  Capabilities  (AC)  and  Performance  ( Perf) 

The  Acquirer  Capability  composite  measure  described  in  Section  5.1.4  showed  a  Moderately 
Strong  negative  relationship  with  the  Project  Performance  composite  measure  defined  in  Section 
5. 1.5.4,  as  shown  in  Figure  58. 
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Figure  58:  Relationship  Between  Acquirer  Capability  and  Performance  ( Perf  Versus  AC) 

The  Acquirer  Capability  was  not  obtained  through  surveying  the  acquirers,  but  was  derived  from 
information  reported  by  the  Suppliers.  The  indirect  nature  of  the  information  may  introduce  addi¬ 
tional  biases  and  errors  compromising  the  analysis. 

Acquirer  Capability  varied  across  the  project  sample  with  15  projects  exhibiting  low  capability, 

19  exhibiting  moderate  capability,  and  12  exhibiting  high  capability.  The  projects  with  purport¬ 
edly  lower  Acquirer  Capability  were  somewhat  more  likely  to  exhibit  Best  Performance.  How- 
ever,  what  is  most  notable  is  the  fact  that  the  projects  with  Higher  Acquirer  Capability  are  much 
more  likely  to  exhibit  Lower  Performance. 

A  Gamma  value  of  -0.35  indicates  that  there  is  a  moderately  strong  negative  relationship  between 
Project  Performance  and  the  elements  of  Acquirer  Capability  addressed  in  this  survey.  A  p  value 
of  0.03  indicates  that  there  is  a  3%  probability  that  a  relationship  of  this  magnitude  could  sponta¬ 
neously  occur  by  chance  alone 

By  themselves,  these  differences  in  Project  Performance  as  a  function  of  Acquirer  Capability  are 
puzzling,  and  no  single  cause  for  this  phenomenon  is  obvious.  However,  bear  in  mind  that  Ac- 
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quirers  were  not  polled  in  this  survey;  so  direct  insight  into  their  capabilities  was  not  possible. 
Rather,  as  noted  in  Section  5.1.4,  the  Acquirer  Capability  composite  measure  is  derived  from  the 
suppliers’  reports  about 

•  the  Acquirer’s  participation  on  Integrated  Project  Teams  (IPTs) 

•  the  Acquirer’s  provision  of  a  Systems  Engineering  Plan  (SEP) 

•  the  Quality  of  system  requirements 

•  the  Completeness  of  system  requirements 

•  the  Stability  of  system  requirements 

Other  things  being  equal,  one  would  think  that  all  of  these  factors  would  contribute  to  greater  pro¬ 
ject  success.  The  problem  is  that  other  things  rarely  are  equal.  Additional  investigation  clearly  is 
needed  to  understand  the  reasons  for  this  counter-intuitive  finding. 

For  now,  we  can  only  make  a  few  reasonably  well  informed  conjectures.  Our  preliminary  analy¬ 
ses  based  on  these  survey  data  are  insufficient  to  present  here.  However,  they  do  suggest  that  Ac¬ 
quirer  Capability  is  fairly  strongly  associated  with  better  SE  Capability.  That  is,  good  acquirers 
are  more  likely  to  select  good  suppliers,  but  better  Acquirer  Capability  appears  to  affect  Project 
Performance  indirectly.  Acquirer  Capability  also  does  seem  to  have  some  mediating  effects  on  the 
nature  of  the  relationships  between  Project  Performance  and  both  Requirements  and  Technical 
Solution  capabilities.  However,  the  relationships  are  much  less  clear-cut  than  are  those  mediated 
by  Project  Challenge  as  shown  in  Section  5.3.1. 

5.3  EFFECTS  OF  PC,  PE,  AND  AC  ON  THE  RELATIONSHIPS  BETWEEN  SEC  AND 
PERF 

In  section  5.2.3,  we  have  examined  the  relationships  between  Project  Performance  and  12  areas  of 
SE  Capabilities.  From  this  we  can  form  some  opinions  regarding  the  value  of  these  SE  practices. 
We  can  also  ask  additional  questions  of  interest  such  as 

•  How  is  the  impact  of  these  SE  practices  upon  Project  Performance  affected  by  the  degree  of 
challenge  in  the  project? 

•  How  is  the  impact  of  these  SE  practices  upon  Project  Performance  affected  by  the  other  fac¬ 
tors  in  the  project  environment? 

•  How  is  the  impact  of  these  SE  practices  upon  Project  Performance  affected  by  the  acquirer’s 
capabilities? 

Unfortunately,  our  data  do  not  allow  us  to  examine  the  mediating  effects  of  Project  Environment 
factors  or  Acquirer  Capabilities.  The  joint  inter-relationships  among  them,  the  SE  Capability 
measures  and  Project  Perfonnance  simply  are  not  sufficiently  well  distributed  in  the  still  relatively 
small  sample  for  this  survey.  There  are  not  enough  projects  in  some  categories  and  too  many  are 
concentrated  in  others.  For  example,  Acquirer  Capability  tends  to  be  higher  when  Project  Chal¬ 
lenge  is  lower.  The  relationship  is  a  relatively  weak  one,  but  it  confounds  the  analysis  nonethe¬ 
less. 

However,  the  response  distributions  are  sufficient  to  permit  an  examination  of  the  mediating  ef¬ 
fects  of  Project  Challenge  on  Project  Performance 
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5.3.1  Moderating  Effects  of  Project  Challenge  (PC) 

To  examine  the  effect  of  Project  Challenge  upon  the  relationships  between  SE  Capabilities  and 
Project  Performance,  we  have  chosen  several  of  the  SE  Capability  areas  that  show  stronger  influ¬ 
ences  on  Project  Performance.  Within  each  of  these  areas,  we  have  partitioned  the  data  set  into 
two  subsets: 

•  those  projects  with  higher  Project  Challenge 

•  those  projects  with  lower  Project  Challenge 

For  each  subset,  as  we  did  for  the  entire  data  set  previously  in  Section  5.2,  we  then  identify  the 
relationship  between  the  SE  Capability  area  and  Project  Performance.  The  SE  Capability  areas 
chosen  for  this  analysis  are 


•  Total  Systems  Engineering  Capability  versus  Performance  ( SEC  versus  Perf) 


Combined  Requirements  and  Technical 
Solution  Capability 


versus 


Performance  ( SECr+ts  versus  Perf) 


There  is  a  good  deal  of  consistency  among  the  comparisons,  despite  the  fact  that  the  number  of 
cases  in  each  subset  is  low.  Such  consistency  is  unlikely  to  have  occurred  by  chance  alone.  The  p 
values  are  lower  than  those  for  the  comparably  strong  bivariate  relationships  reported  for  the  en¬ 
tire  data  set.  However,  that  is  because  of  the  low  numbers  of  cases  in  each  subset  when  we  make 
the  same  comparisons  separately  for  the  higher  and  lower  challenge  projects. 

Since  the  number  of  cases  is  so  small,  one  should  be  especially  careful  not  to  over  interpret  the 
percentage  differences  shown  in  the  figures  in  this  section.  However,  we  do  show  the  percentages 
to  be  consistent  with  the  other  results  in  this  Special  Report. 

5.3. 1.1  Effects  of  Project  Challenge  (PC)  on  the  Relationship  between  Overall  Systems  Engi¬ 
neering  Capability  (SEC)  and  Project  Performance  (Perf) 

The  Overall  Systems  Engineering  Capability  (SEC)  composite  measure  described  in  Section 
5.1.3.13  is  compared  with  the  Project  Performance  (Perf)  composite  measure  defined  in  Section 
5. 1.5. 4  and  controlled  by  the  Project  Challenge  (PC)  composite  measure  of  Section  5.1.1.  The 
results  are  shown  in  Figure  59. 
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Figure  59:  Relationship  Between  Overall  Systems  Engineering  Capability  and  Project  Performance 
(Perf  Versus  SEC)  controlled  by  Project  Challenge  (PC) 

The  strength  of  the  relationship  between  Overall  SE  Capability  and  Project  Performance  is  evi¬ 
dent  for  the  Low  Challenge  projects.  In  this  sample,  of  the  eight  projects  with  lower  Overall  SE 
Capabilities,  25%  showed  Best  performance.  As  Overall  SE  Capabilities  increased  to  moderate 
(five  projects)  to  high  (eight  projects),  achievement  of  Best  Project  Performance  varied  to  20% 
and  75%,  respectively.  Likewise,  as  Overall  SE  Capability  increased  from  Low  to  Moderate  to 
High,  achievement  of  Lower  Project  Performance  decreased  from  38%  to  20%  to  12%,  respec¬ 
tively. 


A  Gamma  value  of  0.55  for  the  Low  Challenge  projects  indicates  that  there  is  a  very  strong  posi¬ 
tive  relationship  between  Project  Performance  and  the  elements  of  Overall  SE  addressed  in  this 
survey.  A  p  value  of  0.02  indicates  that  there  is  a  2%  probability  that  a  relationship  of  this  magni¬ 
tude  could  spontaneously  occur  by  chance  alone. 


A  somewhat  similar  relationship  may  be  seen  for  the  High  Challenge  projects.  In  this  sample,  of 
the  five  projects  with  lower  Overall  SE  Capabilities,  none  showed  Best  performance.  As  Overall 
SE  Capabilities  increased  to  moderate  (twelve  projects)  and  to  high  (eight  projects),  achievement 
of  Best  Project  Performance  increased  to  8%  and  38%,  respectively.  However,  the  differences  in 
Lower  Project  Performance  are  not  consistent.  In  fact,  Lower  Project  Perfonnance  is  most  com¬ 
mon  (50%)  among  the  projects  that  exhibit  Higher  Overall  SE  Capabilities.  As  noted  earlier  in 
Section  5.2.1,  this  may  be  due  to  the  fact  that  the  Overall  SE  Capabilities  measure  is  based  on  all 
of  the  practices  including  those  that  appear  to  have  less  direct  of  an  effect  on  Project  Performance. 

A  Gamma  value  of  0.12  for  the  High  Challenge  projects  indicates  that  there  is  a  weak  positive 
relationship  between  Project  Performance  and  the  elements  of  Overall  Systems  Engineering  ad¬ 
dressed  in  this  survey.  A  p  value  of  0.32  indicates  that  there  is  a  32%  probability  that  a  relation¬ 
ship  of  this  magnitude  could  spontaneously  occur  by  chance  alone. 
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The  comparison  of  Low  Challenge  and  High  Challenge  projects  clearly  shows  that  Systems  Engi¬ 
neering  Capability  has  a  stronger  influence  on  the  Low  Challenge  projects  (Gamma  =  0.55  versus 
Gamma  =  0.12).  One  possible  interpretation  of  this  is  that  the  impact  of  the  Higher  Project  Chal¬ 
lenge  on  Project  Performance  marginalizes  the  impact  of  SE  Capability. 


Further  interpretation  of  these  relationships  is  explored  in  Section  5. 3. 1.3. 

5.3. 1.2  Effects  of  Project  Challenge  (PC)  on  the  Relationship  between  Combined  Require¬ 
ments  and  Technical  Solution  Capability  (SECr+ts)  and  Project  Performance  (Perf) 

The  Combined  Requirements  and  Technical  Solution  Capability  ( SECr+t )  measure  described  in 
Section  5.2.3.14  is  compared  with  the  Project  Performance  ( Perf)  composite  measure  defined  in 
Section  5. 1.5. 4  and  controlled  by  the  Project  Challenge  ( PC)  composite  measure  of  Section  5.1.1 
The  results  are  shown  in  Figure  60. 
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Figure  60:  Relationship  Between  Combined  Requirements/Technical  Solution  Capability  and  Project 
Performance  (Perf  versus  SECR+Ts)  controlled  by  Project  Challenge  (PC) 

The  strength  of  the  relationship  between  combined  Requirements  and  Technical  Solution  capabil¬ 
ity  and  Project  Performance  is  evident  for  the  Low  Challenge  projects.  In  this  sample,  of  the  eight 
projects  with  lower  combined  Requirements  and  Technical  Solution  capabilities,  25%  showed 
Best  performance.  As  combined  Requirements  and  Technical  Solution  capabilities  increased 
through  moderate  (six  projects)  to  high  (seven  projects),  achievement  of  Best  Project  Performance 
rose  to  33%  and  72%,  respectively.  Likewise,  50%  of  the  projects  with  Lower  combined  Re¬ 
quirements  and  Technical  Solution  capability  achieved  Lower  Project  Performance.  In  contrast, 
14%  of  the  high  capability  projects  and  none  of  the  moderate  capability  projects  exhibited  Lower 
Project  Performance. 

A  Gamma  value  of  0.57  for  the  Low  Challenge  projects  indicates  that  there  is  a  very  strong  posi¬ 
tive  relationship  between  Project  Performance  and  the  elements  of  Requirements  and  Technical 
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Solution  addressed  in  this  survey.  A p  value  of  0.02  indicates  that  there  is  a  2%  probability  that 
this  type  of  relationship  could  occur  by  chance  alone. 

A  similar  relationship  may  be  seen  among  the  High  Challenge  projects.  None  of  the  seven  pro¬ 
jects  with  lower  combined  Requirements  and  Technical  Solution  capabilities  and  none  of  the 
seven  projects  with  moderate  capabilities  in  this  sample  achieved  Best  performance.  Yet,  36%  of 
the  1 1  projects  that  exhibited  high  capability  also  did  achieve  Best  Project  Performance.  Simi¬ 
larly,  those  who  achieved  only  Lower  Project  Performance  declined  from  57%  through  43%  to 
27%  respectively. 

A  Gamma  value  of  0.54  for  the  High  Challenge  projects  indicates  that  there  is  a  very  strong  posi¬ 
tive  relationship  between  Project  Performance  and  the  elements  of  Requirements  and  Technical 
Solution  addressed  in  this  survey.  A p  value  of  0.03  indicates  that  there  is  a  3%  probability  that 
this  type  of  relationship  could  occur  by  chance  alone. 

The  comparison  of  Low  Challenge  and  High  Challenge  projects  clearly  shows  that  the  combined 
Requirements  and  Technical  Solution  Capabilities  have  an  equally  strong  influence  on  both  the 
Low  Challenge  and  High  Challenge  projects  (Gamma  =  0.57  versus  Gamma  =  0.54).  Thus,  im¬ 
proved  capabilities  in  the  areas  of  Requirements  and  Technical  Solution  appear  to  have  a  benefi¬ 
cial  impact  upon  Project  Perfonnance,  regardless  of  the  degree  of  Project  Challenge 

Further  interpretation  of  these  relationships  is  explored  in  Section  5. 3. 1.3. 

5.3.1 .3  Summary  of  the  Moderating  Effects  of  Project  Challenge 

The  impact  of  both  Project  Challenge  and  Overall  SE  Capability  is  apparent  in  the  relationships 
explored  in  Sections  5.3. 1.1  and  5. 3. 1.2.  Details  of  this  impact  may  be  seen  in  the  following  ob¬ 
servations: 

•  Referring  to  Figure  59,  for  the  worst  case  scenario  of  lower  Overall  SE  Capability  and  high 
Project  Challenge,  Project  Performance  results  are  discouraging,  with 

-  0%  of  the  projects  reporting  Best  Performance, 

-  60%  reporting  Moderate  Performance,  and 

-  40%  reporting  Lower  Performance. 

A  similar  result  is  seen  in  Figure  60.  When  lower  Requirements  and  Technical  Solution  SE 
Capability  are  applied  to  more  challenging  projects, 

-  0%  of  the  projects  deliver  Best  Performance, 

-  43%  deliver  Moderate  Performance,  and 

-  57%  deliver  Lower  Performance. 

This  clearly  shows  the  risk  of  asking  less  capable  suppliers  to  perform  on  more  challenging 
projects. 

•  Again  referring  to  Figure  59,  within  the  same  group  of  projects  presenting  high  Project  Chal¬ 
lenge,  an  improvement  in  the  Overall  SE  Capability  increases  the  number  of  projects  report¬ 
ing  Best  Performance  from  0%  to  38%.  Likewise,  from  Figure  60  we  see  that  improvement 
in  Requirements  and  Technical  Solution  SE  Capability  increases  the  number  of  projects  re¬ 
porting  Best  Performance  from  0%  to  36%.  This  clearly  shows  the  value  of  Overall  SE  Ca¬ 
pability  in  addressing  challenging  projects. 
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•  As  seen  in  Figure  59,  for  high  Project  Challenge  projects,  a  significant  percentage  of  projects 
deliver  Lower  Performance  (33  to  50%)  regardless  of  the  Overall  SE  Capability.  For  the  low 
Project  Challenge  projects,  the  percentage  of  projects  delivering  Lower  Performance  ranges 
from  12  to  30%  -  a  significant  improvement.  Similarly  from  Figure  60,  for  the  high  challenge 
projects  the  percentage  of  projects  delivering  Lower  Performance  ranges  from  27  to  57%. 

For  less  challenging  projects,  this  range  drops  to  14  to  50%.  This  clearly  shows  the  impact  of 
Project  Challenge  on  Project  Performance,  and  suggests  that  improving  Overall  SE  Capability 
is  not  enough;  Project  Challenge  must  also  be  managed. 

•  Once  again  from  Figure  59,  for  the  best  case  scenario  of  Fligher  Overall  SE  Capability  and 
low  Project  Challenge,  Project  Performance  results  are  very  favorable,  with  75%  of  less  chal¬ 
lenging  projects  reporting  Best  Performance  and  only  12%  reporting  Lower  Performance. 
Likewise  from  Figure  60,  72%  of  less  challenging  projects  reported  Best  Performance  and  on¬ 
ly  14%  reported  Lower  Performance.  This  illustrates  the  ideal  situation  that  is  attainable 
through  improvement  of  SE  Capabilities  and  reduction  and  management  of  Project  Challenge. 

As  noted  previously,  we  were  unable  to  do  rigorous  multivariate  statistical  analyses  of  the  com¬ 
bined  effect  of  several  measures  on  Project  Performance  ( Perf)  because  of  the  small  number  of 
project  responses.  Instead  we  have  created  a  new  overall  measure  by  using  the  statistical  relation¬ 
ships  between  the  Combined  Requirements  and  Technical  Solution  Capability  (SECr+ts)  measure 
described  in  Section  5.2.3.14  and  the  Project  Challenge  ( PC)  measure  described  in  Section  5.1.1. 
As  usual,  three  categories  were  used  since  a  relatively  small  number  of  projects  participated  in  the 
survey.  Scoring  was  as  follows:7 

•  Projects  that  exhibited  both  higher  SECr+ts  capability  and  lower  PC  challenge  were  catego¬ 
rized  in  Figure  61  as  “Higher  Capability  and  Lower  Challenge”.  So  too  were  projects  that 
scored  in  one  of  those  same  two  categories  along  with  the  middle  category  on  the  other. 

•  Similarly,  projects  that  exhibited  both  lower  SECr+ts  capability  and  higher  PC  challenge  were 
categorized  in  Figure  61  as  “Lower  Capability  and  Higher  Challenge”.  So  too  were  projects  that 
scored  in  one  of  those  same  two  categories  along  with  the  middle  category  on  the  other. 

•  Projects  that  exhibited  both  moderate  capability  and  moderate  challenge  were  categorized  as 
“Mixed  Capability  and  Challenge”.  So  too  were  projects  that  exhibited  both  lower  SECr+ts 
capability  and  lower  PC  challenge,  as  were  those  that  exhibited  both  higher  SECr+ts  capabil¬ 
ity  and  higher  PC  challenge. 

This  categorization  is  illustrated  in  Table  5. 


7  The  cutting  points  for  PC  were  changed  somewhat  to  create  a  better  balanced  joint  distribution  for  the  new  higher 
order  SECR+Ts+PC  composite  measure.  The  adjusted  cutting  points  for  PC  are  1 .76  and  2.04. 
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Table  5:  SE  Capability  and  Project  Challenge  Categorization 


SECr+ts  Capability  | 
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Project 
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1  Lower  Capability  and  Higher 

&  Capability 

Moderate 

Challenge 

Challenge 

Higher  1 
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Capability  &  Lower  Challenge  | 

This  Combined  Requirements/Technical  Solution  Capability  and  Project  Challenge 
(SECr+ts+PC)  measure  was  compared  with  the  Project  Performance  ( Perf)  composite  measure 
defined  in  Section  5. 1.5.4.  The  results  are  shown  in  Figure  61. 
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Figure  61:  Relationship  Between  Combined  Requirements/Technical  Solution  Capability  and  Project 
Challenge  on  Project  Performance  (Perf  Versus  SECr+ts+PC) 

The  overall  effects  of  Systems  Engineering  Capability  combined  with  Project  Challenge  varied 
across  the  survey  sample  with  12  projects  exhibiting  Lower  Capability  and  Higher  Challenge,  20 
exhibiting  Mixed  Capability  and  Challenge,  and  14  exhibiting  Higher  Capability  and  Lower  Chal¬ 
lenge.  A  very  strong  0.63  positive  relationship  between  SECr+ts+PC  and  Perf  is  evident.  Half 
(50%)  of  the  projects  that  exhibited  Higher  Capability  and  faced  lower  Project  Challenge 
achieved  Best  Perfonnance.  However,  none  of  the  projects  that  exhibited  Lower  Capability  and 
faced  Higher  Challenge  managed  to  achieve  Best  Perfonnance.  Similarly,  67%  of  the  projects 
with  Lower  Capability  and  Higher  Challenge  managed  only  to  achieve  Lower  Performance.  Only 
14%  of  projects  with  Higher  Capability  and  Lower  Challenge  exhibited  Lower  Performance. 
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A  Gamma  value  of  0.63  indicates  that  there  is  a  very  strong  relationship  between  Project  Per¬ 
formance  and  the  aspects  of  Systems  Engineering  Capability  combined  with  Project  Challenge 
addressed  in  this  survey.  A  p  value  of  0.0004  indicates  that  there  is  an  extremely  low  probability 
that  this  type  of  relationship  could  occur  by  chance  alone. 

An  alternative  way  to  visualize  these  relationships  is  shown  in  Figure  62  and  Figure  63.  In  Figure 
60,  the  two  graphs  contain  six  columns,  each  with  percentages  of  projects  exhibiting  Best,  Moder¬ 
ate,  and  Fower  Project  Performance.  For  each  column,  we  can  calculate  a  performance  score 
through  the  weighted  combination  of  these  three  percentages.  This  score  is  calculated  as: 

Score  =  '/>  x  [Ox  (%  of  Fower  Performance  projects)  + 

1  x  (%  of  Moderate  Performance  Projects)  + 

2  x  (%  of  Best  Performance  Projects)  ] 

This  score  gives  a  weight  of  2  to  projects  exhibiting  Best  Performance,  a  weight  of  1  to  projects 
exhibiting  Moderate  Performance,  and  a  weight  of  0  to  projects  exhibiting  Fower  Performance. 
The  factor  of  Vi  included  in  the  formula  normalizes  the  score  to  range  from  0  (i.e.  100%  of  the 
projects  exhibit  Fower  Performance)  to  1  (i.e.,  100%  of  the  projects  exhibit  Best  Performance). 

Figure  62  and  Figure  63  illustrate  this  score  as  a  function  of  Project  Challenge  and  SE  Capabili¬ 
ties. 
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Figure  62:  Performance  vs.  Project  Challenge  and  Requirements  +  Technical  Solution  SE  Capability 
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Figure  63:  Performance  vs.  Project  Challenge  and  Overall  SE  Capability 

Both  figures  clearly  show  the  combined  impacts  of  Project  Challenge  and  SE  Capability. 
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6  Limitations  and  Lessons  Learned 


Development  and  execution  of  this  survey  was  a  complex  effort.  Complex  efforts  produce  chal¬ 
lenges.  Challenges  produce  learning.  Some  of  the  lessons  learned  from  this  activity  follow. 

1 .  The  lack  of  a  widely  accepted  definition  of  Systems  Engineering  need  not  be  an  obstacle 
to  identifying  SE  activities  deployed  on  projects. 

The  debate  over  the  definition  of  Systems  Engineering  has  raged  for  decades,  without  resolu¬ 
tion.  Even  today,  there  is  no  widely  accepted  definition,  flow  do  you  survey  a  capability  if 
you  cannot  define  that  capability?  This  question  had  thwarted  previous  attempts  of  the  SEEC 
to  identify  the  value  of  SE.  This  survey  avoided  that  obstacle.  Rather  than  enter  the  debate  as 
to  what  does  or  does  not  constitute  SE,  this  survey  examined  specific  activities  and  practices 
that  were  likely  to  be  considered  as  elements  of  SE  by  most  systems  engineers.  Decisions  as 
to  what  to  include  in  the  survey  and  what  to  omit  were  made  by  a  panel  of  experienced  sys¬ 
tems  engineers. 

2.  Indirect  access  to  respondents  helped  protect  confidentiality  of  respondent’s  data,  but 
made  it  more  difficult  to  track  progress  and  to  analyze  data 

The  questionnaire  requested  a  great  amount  of  information  about  the  methods  deployed  on  a 
project  (i.e.,  the  Systems  Engineering  activities  applied  to  the  project)  and  the  results 
achieved  by  the  project  (i.e.,  the  Project  Performance).  Exposure  of  this  information  to  com¬ 
petitors  and/or  customers  could  be  disadvantageous  to  the  responding  project  or  organization, 
exposing  operational  details  in  a  competitive  environment.  For  this  reason,  the  SEEC  made  a 
decision  early  in  the  development  of  the  survey  that  respondent’s  data  must  be  securely  han¬ 
dled.  This  decision  drove  a  number  of  subsequent  decisions: 

a.  The  SEI  was  chosen  as  a  trusted  agent  to  collect,  analyze,  and  report  the  survey  re¬ 
sults. 

b.  The  questionnaire  was  constructed  to  collect  no  information  that  could  expose  the 
identity  of  the  respondent,  the  project,  or  the  organization. 

c.  The  survey  execution  was  done  via  the  Web,  enabling  the  collection  of  responses  in 
an  anonymous  manner. 

d.  Respondents  were  solicited  via  proxy,  using  “focal  contacts”  within  targeted  organi¬ 
zations  to  identify,  solicit,  and  expedite  respondents  within  that  organization. 

Of  these  decisions,  the  last  proved  to  be  the  most  problematic.  The  use  of  proxies  isolated  the 
survey  execution  team  from  the  respondents,  as  it  was  intended  to  do.  However,  this  meant 
that  the  survey  execution  was  largely  dependent  on  the  efforts  of  the  proxies.  These  proxies 
were  asked  to 

-  identify  candidate  respondents 

-  solicit  the  participation  of  the  respondents 

-  report  the  number  of  respondents  solicited  to  the  SEI 
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-  expedite  respondents  at  defined  times,  and  report  the  number  of  responses  submitted 
to  the  SEI 

The  first  challenge  encountered  by  the  SEEC  was  the  identification  of  appropriate  and  willing 
proxies.  The  SEEC  initially  attempted  a  top-down  approach,  contacting  candidate  proxies  at 
the  highest  levels  of  the  organization,  and  asking  them  to  propagate  the  survey  throughout  the 
organization.  This  approach  was  only  partially  successful.  In  some  cases,  the  candidate  prox¬ 
ies  were  not  aware  of  or  aligned  with  the  mission  of  the  SEEC  and  NDIA  in  general.  The 
SEEC  was  unable  to  convince  them  of  the  value  of  this  survey.  In  some  cases,  these  candidate 
proxies  were  just  too  busy  to  address  this  request.  For  cases  where  the  SEEC  was  unsuccess¬ 
ful  at  penetrating  an  organization  at  the  top  level,  we  resorted  to  lower  levels — rather  than 
work  through  corporate  headquarters,  we  attempted  to  contact  proxies  at  the  divisional  level. 
This  again  was  only  partially  successful.  Again,  in  some  cases,  the  candidate  proxies  were  not 
aware  of  or  aligned  with  the  mission  of  the  SEEC  and  NDIA  in  general.  The  SEEC  was  un¬ 
able  to  convince  them  of  the  value  of  this  survey.  In  some  cases,  again,  these  candidate  prox¬ 
ies  were  just  too  busy  to  address  this  request.  In  yet  other  cases,  the  candidate  proxies  were 
unwilling  to  proceed  without  corporate  approval. 

Once  the  proper  proxies  were  found,  the  SEEC  then  became  dependent  on  their  efforts  to  ex¬ 
ecute  the  survey.  While  the  SEEC  carefully  crafted  instructions  and  solicitation  aids  for  the 
proxies  (see  APPENDIX  C),  the  initiative  to  identify  and  solicit  respondents  fell  wholly  to  the 
proxies.  While  many  did  a  fine  job,  others  did  not.  Due  to  the  anonymous  nature  of  the  survey 
responses,  the  SEEC  has  no  way  of  knowing  who  did  or  did  not  respond.  However,  we  do 
know  that  many  of  the  proxies  failed  to  provide  the  oft-requested  response  rate  data;  thereby 
implying  little  or  no  response. 

The  use  of  proxies  was  intended  to  increase  the  response  rate  by 

a.  enhancing  the  anonymity  of  the  respondents 

b.  enhancing  the  credibility  of  the  survey  effort  within  the  responding  organization  by 
having  the  solicitation  of  respondents  done  by  insiders 

While  the  use  of  proxies  may  have  aided  in  the  achievement  of  these  goals,  success  was  com¬ 
promised  by  the  challenge  of  the  indirect  contact  between  the  researchers  and  the  respon¬ 
dents.  The  results  were  fewer  responses  than  desired.  We  had  hoped  for  at  least  200  re¬ 
sponses;  we  received  64.  This  limited  our  ability  to  do  more  detailed  statistical  analyses  of 
several  important  topics.  Equally  importantly,  we  were  not  able  to  generalize  some  of  our 
findings  more  widely  because  the  sample  was  not  based  explicitly  on  known  probabilities  of 
selection  [Foreman  1991].  Although  it  is  impossible  to  know  whether  the  number  of  re¬ 
sponses  would  have  increased  with  an  alternate  solicitation  method,  this  is  a  topic  for  further 
consideration  in  future  surveys. 

3.  Piloting  by  the  SEEC  did  not  fully  expose  question  interpretation  issues 

As  noted  in  Section  3.1,  the  survey  execution  methods  and  the  survey  questionnaire  were 
tested  in  a  pilot  phase.  The  questionnaire  was  distributed  to  a  small  number  of  members  of  the 
SEEC,  who  were  requested  to  respond  using  the  web  site  developed  for  the  survey.  This  ac¬ 
tivity  provided  a  number  of  useful  outcomes: 
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a.  It  revealed  that  the  survey  was  too  long.  Initial  times  to  complete  the  questionnaire 
ranged  from  30  minutes  to  3  hours. 

b.  It  exposed  a  number  of  issues  with  the  Web-based  data  collection  process. 

As  a  result  of  this  piloting,  the  questionnaire  was  substantially  reduced  and  simplified,  reduc¬ 
ing  the  average  completion  time  to  about  40  minutes.  The  issues  with  the  Web-based  collec¬ 
tion  process  were  also  addressed. 

In  retrospect,  it  appears  that  the  piloting,  while  helpful,  was  not  sufficient  to  expose  additional 
weaknesses  in  the  questionnaire.  Only  after  the  responses  were  received  did  it  become  appar¬ 
ent  that  a  few  of  the  questions  were  not  as  clearly  stated  as  they  needed  to  be.  This  was  evi¬ 
dent  from  the  unreasonably  wide  range  of  responses  to  some  questions,  and  from  inconsisten¬ 
cies  with  these  responses  and  responses  to  other  questions  in  the  survey.  These  issues  only 
affected  a  few  questions  within  the  survey.  As  a  result,  the  responses  to  these  questions  were 
not  used  in  the  overall  analysis. 

Two  reasons  are  postulated  for  the  inability  of  the  pilot  effort  to  recognize  these  weaknesses. 

a.  The  number  of  respondents  involved  in  the  piloting  was  too  small. 

b.  The  pilot  respondents  may  not  have  been  representative  of  the  survey  population.  The 
pilot  respondents  were  self-selected  from  the  members  of  the  SEEC.  As  such,  they  were 
perhaps  more  knowledgeable  about  Systems  Engineering  than  the  average  respondent. 
Additionally,  they  may  have  been  more  motivated  than  the  average  respondent. 

4.  Insufficient  attention  to  the  adequacy  of  survey  sampling  analysis  methods  during  sur¬ 
vey  development 

Prior  to  the  development  of  the  survey,  the  survey  main  hypothesis  (i.e.,  effective  perform¬ 
ance  of  Systems  Engineering  best  practices  results  in  quantifiable  improvement  in  program 
execution)  was  defined  (see  Section  1.3).  The  questionnaire  was  crafted  to  test  this  hypothe¬ 
sis.  Similarly,  the  other  survey  questions  were  crafted  to  allow  us  to  test  more  detailed  hy¬ 
potheses  about  the  mediating  effects  of  other  pertinent  factors  that  might  affect  both  the  use 
of  SE  best  practices  and  Project  Perfonnance  under  varying  conditions.  However  a  number  of 
uncertainties  remained  unresolved  throughout  the  development  of  the  survey.  In  particular, 
we  were  unsure  about  the  number  and  types  of  responses  that  would  be  received.  Responses 
could  have  come  from  $50,000  sub  contract  efforts  or  billion  dollar  programs.  While  the  data 
analysis  methods  to  be  used  on  the  responses  were  known,  we  were  reluctant  to  commit  to  a 
firm  analysis  plan  until  some  of  this  uncertainty  was  resolved. 

As  is  common  in  exploratory  data  analysis,  a  detailed  analysis  plan  was  not  fonned  until  after 
the  responses  were  received.  The  analysis  was  then  perfonned  iteratively  [Tukey  1977].  As 
noted  previously,  fewer  responses  were  received  than  hoped  for.  Also,  the  sizes  and  kinds  of 
responding  projects  varied  over  a  large  range.  These  factors  did  indeed  influence  the  analysis 
plan,  with  the  smaller-than-desired  number  of  responses  limiting  the  number  and  kinds  of 
analyses  that  could  be  performed. 

Nevertheless,  the  development  of  a  more  detailed  analysis  plan  prior  to  the  deployment  of  the 
questionnaire  would  have  been  helpful.  In  completing  the  analysis  plan,  several  instances 
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were  encountered  where  additional  information  would  have  been  useful  in  understanding  the 
project’s  responses.  Had  these  needs  been  determined  prior  to  deployment,  the  questionnaire 
could  have  been  modified  to  collect  the  needed  data. 

5.  Insufficient  stakeholder  involvement  limited  response  to  the  survey 

Throughout  the  development  and  execution  of  this  survey,  the  SEEC  made  significant  efforts 
to  involve  all  stakeholders.  All  SEEC  meetings  were  open  to  all  members  of  the  NDIA  SED. 
Anyone  who  attended  a  meeting  was  considered  a  committee  member  and  was  placed  on  the 
committee  email  list — a  list  that  grew  to  54  names.  Weekly  or  biweekly  telephone  confer¬ 
ences  were  held.  Status  reports  were  presented  at  SED  meetings.  In  spite  of  these  efforts,  in¬ 
volvement  of  some  stakeholders  remained  insufficient.  Future  efforts  need  to  ensure  closer 
coordination  to  ensure  continued  stakeholder  involvement. 
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7  Summary 


The  impetus  for  this  survey  was  a  desire  to  answer  the  questions: 

1 .  What  will  the  application  of  Systems  Engineering  practices  cost  me? 

2.  What  benefits  will  I  gain  from  the  application  of  these  practices? 

An  understanding  of  these  answers  is  needed  to  justify  a  project’s  investment  in  SE  resources  and 
activities.  To  address  these  questions,  we  assessed  the  impact  of  the  deployment  of  SE  practices 
on  Project  Performance.  Knowing  that  SE  was  not  the  only  factor  influencing  Project  Perform¬ 
ance,  we  also  assessed  Project  Challenge,  Project  Environment  factors,  and  Acquirer  Capability  to 
identify  their  relationship  to  Project  Performance.  The  analysis  of  the  collected  data  shows  that 
there  are  indeed  identifiable  relationships  between  many  of  these  driving  factors  and  Project  Per¬ 
formance.  Ranked  by  the  strength  of  association  with  Project  Performance,  these  driving  factors 
are  shown  in  Table  6. 


Table  6:  Ranked  Project  Performance  Driving  Factors 


Driving  Factor8 

Type 

Relationship  to  Project  Performance 

Section 

Reference 

Description 

(Gamma9) 

Requirements  and  Technical 

Solution  Combined  with  Project  Chal¬ 
lenge 

SEC 

+PC 

Very  strong  positive 

+0.63 

5.3. 1.3 

Combined  Requirements  and  Technical 
Solution 

SEC 

Strong  positive 

+0.49 

5.2.3.14 

Product  Architecture 

SEC 

Moderately  strong  to  strong  posi¬ 
tive 

+0.40 

5.1. 3.7 

Trade  Studies 

SEC 

Moderately  strong  to  strong  posi¬ 
tive 

+0.37 

5.1. 3.6 

IPT-Related  Capability 

SEC 

Moderately  strong  positive 

+0.34 

5.1. 3.1 

Technical  Solution 

SEC 

Moderately  strong  positive 

+0.36 

5.1. 3.8 

Requirements  Development 
and  Management 

SEC 

Moderately  strong  positive 

+0.33 

5.1. 3.5 

Overall  Systems  Engineering  Capability 

SEC 

Moderately  strong  positive 

+0.32 

5.1.3.13 

Use  caution  in  to  avoid  over-interpreting  the  meaning  of  the  Systems  Engineering  Capability  (SEC)  categories, 
Project  Challenge,  and  Project  Environment  Factors  listed  in  Table  6.  For  example,  the  “Project  Planning”  cate¬ 
gory  does  include  elements  of  project  planning,  but  is  not  a  comprehensive  compilation  of  all  project  planning  ac¬ 
tivities.  To  properly  understand  the  listed  relationships,  please  refer  to  the  report  sections  listed  in  the  last  column 
to  see  what  survey  questions  are  included  in  the  SEC  category. 

9  Gamma  is  a  measure  of  association  that  expresses  the  strength  of  relationship  between  two  ordinal  variables, 
with  values  near  -1  indicating  a  strong  opposing  relationship,  values  near  0  indicating  a  weak  or  no  relationship 
(statistical  independence),  and  values  near  +1  indicating  a  strong  supporting  relationship 
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Table  6:  Ranked  Project  Performance  Driving  Factors 


Driving  Factor8 

Type 

Relationship  to  Project  Performance 

Section 

Reference 

Description 

(Gamma9) 

Project  Challenge 

PC 

Moderately  strong  negative 

-0.31 

5.1.1 

Validation 

SEC 

Moderately  strong  positive 

+0.28 

5.1.3.11 

Risk  Management 

SEC 

Moderately  strong  positive 

+0.28 

5.1. 3.4 

Verification 

SEC 

Moderately  strong  positive 

+0.25 

5.1.3.10 

Product  Integration 

SEC 

Weak  positive 

+0.21 

5.1. 3.9 

Project  Planning 

SEC 

Weak  positive 

+0.13 

5.1. 3.2 

Configuration  Management 

SEC 

Weak  positive 

+0.13 

5.1.3.12 

Process  Improvement 

PE 

Weak  positive 

+0.05 

5.1 .2.3 

Project  Monitoring  and  Control 

SEC 

Weak  negative 

-0.13 

5.1. 3. 3 

The  survey  also  examined  Project  Environment  factors  that  may  or  may  not  influence  Project  Per¬ 
formance.  Due  to  the  relatively  small  sample  size  and  the  small  number  of  respondents,  the  num¬ 
ber  of  projects  in  each  answer  category  for  the  Project  Environment  questions  were  sufficiently 
small  to  reduce  the  confidence  one  can  have  in  these  findings.  Results  are  presented  in  this  report, 
but  care  should  be  taken  not  to  over  interpret  these  differences. 

Finally,  the  survey  examined  the  impact  of  the  Acquirer’s  capabilities  upon  Project  Performance. 
Although  the  survey  was  not  specifically  designed  to  provide  a  detailed  assessment  of  the  ac¬ 
quirer’s  capabilities,  some  responses  from  the  suppliers  could  be  used  to  develop  a  rudimentary 
relative  measure  of  some  acquirer  capabilities.  Due  to  the  narrow  scope  of  the  acquirer  assess¬ 
ment,  and  the  indirect  nature  of  this  assessment  (i.e.,  assessment  of  acquirers  via  suppliers),  the 
relationships  between  Acquirer  Capability  and  Project  Performance  are  unclear. 

The  moderately  strong  statistical  relationships  between  Systems  Engineering  Capabilities  and 
Project  Performance  summarized  here  are  notable  by  themselves.  Other  things  being  equal,  better 
Systems  Engineering  Capabilities  do  tend  to  lead  to  better  Project  Performance.  Of  course.  Sys¬ 
tems  Engineering  Capability  alone  does  not  ensure  outstanding  Project  Performance.  The  survey 
results  also  show  notable  differences  in  the  relationship  between  SE  best  practices  and  perform¬ 
ance  among  more  challenging  as  compared  to  less  challenging  projects  (section  5.3.1). 

Table  7  provides  a  summary  of  the  relationships  analyzed  in  Section  5.2.  Each  row  of  the  table 
shows  a  parameter  (e.g.,  Project  Challenge,  an  SE  Capability)  whose  relationship  to  Project  Per¬ 
formance  was  analyzed  in  Section5.2.  The  columns  of  the  table  show: 

•  the  break  points  defining  the  Lower,  Moderate,  and  Higher  categories  of  each  parameter. 

•  the  percentage  of  Lower  Performance,  Moderate  Performance,  and  Best  Performance  projects 
contained  within  each  category 

•  the  ‘Gamma’  (T)  and  ‘p’  statistics  calculated  for  each  relationship. 
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Table  7;  Summary  of  Relationship  Data 


Lower  Category 

% 

% 

% 

Min. 

Low 

Mod 

Best 

Max. 

Range 

Perf 

Perf 

Perf 

Range 

Project  Challenge 

PC  |  1.0 


Project  Environment 

CMMI 
IMP 
EXP 


Moderate  Category 

% 

% 

% 

Min. 

Low 

Mod 

Best 

Max. 

Range 

Perf 

Perf 

Perf 

Range 

22%  28%  50%  1.85 


][ 


1.0 

36% 

57% 

7% 

1.95 

1.0 

25% 

55% 

20% 

2.17 

1.0 

29% 

42% 

29% 

2.5 

Higher  Category 

% 

% 

% 

Min. 

Low 

Mod 

Best 

Max. 

Range 

Perf 

Perf 

Perf 

Range 

1.85  42%  58%  0%  2.05 


][ 


1.95 

29% 

36% 

35% 

2.7 

2.17 

42% 

29% 

29% 

2.84 

2.5 

39% 

44% 

17% 

3.5 

2.05 


38%  38%  25%  4.0 


][ 


2.7 

33% 

28% 

39% 

4.0 

2.84 

33% 

25% 

42% 

4.0 

3.5 

29% 

29% 

42% 

4.0 

r 

P 

-31% 

5.0% 

22% 

5% 

9% 

13.0% 

39.0% 

33.0% 

Systems  Engineering  Capability 
IPT 
PP 
PMC 
RSKM 
REQ 
TRADE 
ARCH 
TS 
PI 

VER 
VAL 
CM 

Overall  SEC 
REQ+TS 


Acquirer  Capability _  _  _  _ 

AC  |  1.0  ||  7%  60%  33%  ||  2.5  1  |  2.5  ||  41%  32%  26%  ||  3.0  1  |  3.0  ||  50%  25%  25%  ||  4.0  1  |  -35%  3.0%] 


2.5 

43% 

38% 

19% 

3.1 

2.8 

29% 

35% 

36% 

3.3 

2.5 

23% 

46% 

31% 

3.0 

2.8 

27% 

66% 

7% 

3.6 

2.8 

26% 

53% 

21% 

3.4 

2.7 

42% 

41% 

17% 

3.3 

2.7 

29% 

42% 

29% 

3.3 

2.8 

33% 

40% 

27% 

3.2 

1.5 

33% 

38% 

29% 

3.5 

2.7 

33% 

34% 

33% 

3.2 

2.7 

17% 

66% 

17% 

3.3 

3.0 

46% 

36% 

18% 

3.67 

2.5 

29% 

59% 

12% 

3.0 

2.8 

23% 

62% 

15% 

3.1 

1.0 

33% 

54% 

13% 

2.5 

1.0 

33% 

54% 

13% 

2.8 

1.0 

23% 

54% 

23% 

2.5 

1.0 

35% 

47% 

18% 

2.8 

1.0 

44% 

38% 

18% 

2.8 

1.0 

39% 

44% 

17% 

2.7 

1.0 

45% 

44% 

11% 

2.7 

1.0 

40% 

53% 

7% 

2.8 

1.0 

36% 

54% 

14% 

1.5 

1.0 

31% 

62% 

7% 

2.7 

1.0 

54% 

23% 

23% 

2.7 

1.0 

29% 

47% 

24% 

3.0 

1.0 

39% 

46% 

15% 

2.5 

1.0 

43% 

50% 

13% 

2.8 

3.1 

20% 

27% 

53% 

4.0 

3.3 

35% 

29% 

36% 

4.0 

3.0 

45% 

25% 

30% 

4.0 

3.6 

36% 

0% 

64% 

4.0 

3.4 

27% 

18% 

55% 

4.0 

3.3 

19% 

32% 

49% 

4.0 

3.3 

23% 

31% 

46% 

4.0 

3.2 

27% 

27% 

46% 

4.0 

3.5 

29% 

29% 

42% 

4.0 

3.2 

33% 

20% 

47% 

4.0 

3.3 

29% 

33% 

38% 

4.0 

3.67 

28% 

33% 

39% 

4.0 

3.0 

31% 

13% 

56% 

4.0 

3.1 

22% 

28% 

50% 

4.0 

34% 

4.0% 

13% 

25.0% 

-13% 

25.0% 

28% 

6.1% 

33% 

4.0% 

37% 

3.0% 

40% 

0.2% 

36% 

3.0% 

21% 

16.0% 

25% 

9.0% 

28% 

7.0% 

13% 

26.0% 

32% 

4.0% 

49% 

0.5% 
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In  summary,  both  Systems  Engineering  and  Project  Challenge  must  be  considered  together  in  ex¬ 
plaining  variation  in  Project  Performance.  Just  as  higher  Systems  Engineering  Capability  is  asso¬ 
ciated  with  better  Project  Performance,  higher  Project  Challenge  is  associated  with  lower  Project 
Performance.  It  is  the  combination  of  capability  and  challenge  that  better  explains  the  variation  in 
performance  than  does  either  one  alone.  This  is  illustrated  in  Figure  64. 


Figure  64:  Performance  vs.  Project  Challenge  and  Overall  SE  Capability 

As  shown  in  this  report,  improving  Systems  Engineering  capabilities  clearly  can  result  in  better 
Project  Performance.  However,  more  consideration  also  must  be  paid  to  ways  of  reducing  Project 
Challenge.  Doing  so  is  a  major  challenge  prior  to  the  establishment  of  the  development  project, 
beginning  during  the  pre-acquisition  period.  Earlier  application  of  Systems  Engineering  practices 
and  principles  may  go  a  long  way  towards  reducing  that  challenge. 

The  relationships  presented  herein  may  be  used  in  a  number  of  ways: 

•  Defense  contractors  can  use  this  report  to  plan  capability  improvement  efforts  for  their  SE 
programs.  By  focusing  improvement  resources  on  those  SE  activities  most  strongly  associ¬ 
ated  with  improved  Project  Performance,  management  may  optimize  the  efficiency  and  effec¬ 
tiveness  of  those  improvement  efforts. 

•  Defense  contractors  can  compare  their  organization’s  SE  performance  against  the  industry 
benchmark  established  by  this  survey.  Projects  within  the  organization  can  complete  the  sur¬ 
vey  questionnaire.  Their  responses  can  then  be  compared  against  the  aggregated  survey  re¬ 
sponses  question-by-question  to  get  a  measurement  of  the  project’s  SE  Capability  relative  to 
the  survey  population.  This  benchmarking  process  can  be  periodically  repeated  to  track  the 
impact  of  SE  improvement  efforts 

Note  that  the  question-by-question  responses  are  contained  in  APPENDIX  D.  As  promised. 
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the  survey  participants  will  have  access  to  this  information  upon  the  limited  publication  and 
distribution  of  this  report.  Others  will  not  have  access  to  this  data  until  the  report  is  publicly 
released  one  year  later. 

•  Systems  Engineers  and  SE  managers  at  defense  contractors  can  use  this  report  as  justification 
for  and  in  defense  of  their  SE  estimates. 

•  Acquisition  PMOs  may  use  this  report  to  plan  contractor  evaluations  during  RFP  develop¬ 
ment  and  source  selection.  Since  this  survey  shows  clear  statistical  relationships  between  spe¬ 
cific  SE  Capabilities  and  improved  Project  Performance,  acquirers  can  structure  RFPs  and 
source  selection  activities  to  include  evaluation  and  consideration  of  these  capabilities;  there¬ 
by  increasing  the  likelihood  of  project  success. 

•  Throughout  the  execution  of  a  project,  acquisition  PMOs  may  employ  this  survey  or  similar 
methods  to  collect  data  from  suppliers  as  a  means  of  identifying  supplier  deficiencies  contrib¬ 
uting  to  project  risks. 

•  OSD  may  use  this  survey  as  guidance  to  Project  Managers  conveying  the  value  of  SE.  This 
knowledge  can  assist  the  PMs  in  prioritizing  resources  and  evaluating  supplier  budgets. 
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8  Next  Steps 


This  report  shows  a  clear  relationship  between  the  deployment  of  SE  practices  and  improved  Pro¬ 
ject  Performance.  While  the  output  of  this  survey  activity  is  complete  in  its  current  form,  and 
needs  no  further  work  to  find  application  in  the  defense  industry  today,  it  also  suggests  some  di¬ 
rections  for  future  initiatives. 

8.1  CORRELATE  REPORT  FINDINGS  WITH  OTHER  SOURCES 

While  numerous  other  initiatives  have  been  undertaken  within  government,  industry,  and  acade¬ 
mia  to  characterize  the  best  SE  practices  (or  lack  thereof)  applied  on  defense  programs,  the  benefit 
of  this  report  is  to  support  informed  decision-making  based  on  both  quantitative  and  qualitative 
measures  of  effectiveness.  This  data  can  be  used  to  complement  findings  derived  from  other 
sources,  such  as  studies,  reports,  or  root  cause  analyses  of  program  performance  issues,  to  develop 
a  well-rounded  picture  of  the  state -of-the -practice  for  SE  within  the  defense  industry  and  to  priori¬ 
tize  improvement  actions  in  areas  that  are  likely  to  have  the  greatest  benefit  in  improved  program 
performance. 

8.2  DEVELOP  IMPROVEMENT  RECOMMENDATIONS 

Based  on  the  findings  of  this  report,  the  NDIA  Systems  Engineering  Effectiveness  Committee 
(SEEC)  will  develop  recommendations  for  government  and  industry  actions  needed  to  improve 
the  practice  of  systems  engineering  on  DoD  programs. 

Candidate  areas  for  these  recommendations  may  include,  but  are  not  limited  to 

•  updates  to  OSD  policy  and  guidance  to  reinforce  the  application  and  support  of  sound  sys¬ 
tems  engineering  practices  on  programs 

•  improved  training  in  targeted  SE  capability  areas  (significant  strengths  or  weaknesses) 

•  recommendations  on  standard  measures  to  be  collected  and  analyzed 

•  suggested  improvement  to  evaluation  criteria  for  program  plans,  reviews,  or  risk  analyses 

•  greater  communication  of  proven  SE  best  practices  (e.g.,  publications,  conferences,  work¬ 
shops) 

Note  that  numerous  other  efforts  are  already  underway  to  improve  systems  engineering  capabili¬ 
ties  across  the  defense  industrial  base.  For  example,  in  the  Office  of  the  Deputy  Under  Secretary 
of  Defense  (A&T),  the  Director,  Systems  and  Software  Engineering  has  established  a  number  of 
initiatives  focusing  on  SE  [Schaeffer  2007].  These  include 

•  issuing  a  Department- wide  Systems  Engineering  (SE)  policy 

•  issuing  guidance  on  SE,  T&E,  and  SE  Plans  (SEPs) 

•  integrating  DT&E  with  SE  policy  and  assessment  functions — focusing  on  effective,  early 
engagement  of  both 
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•  working  with  Defense  Acquisition  University  to  revise  curricula  (SPRDE,  T&E,  PQM,  LOG, 
PM,  ACQ,  FM,  CONT) 

•  establishing  the  SE  Forum  to  ensure  senior-level  focus  within  DoD 

•  leveraging  close  working  relationships  with  industry  and  academia 

•  instituting  system-level  assessments  in  support  of  DAB,  OIPT,  DAES,  and  in  support  of  pro¬ 
grams 

•  instituting  a  renewed  emphasis  on  modeling  and  simulation  in  acquisition 

To  maximize  the  likelihood  of  positive  action,  recommendations  developed  by  the  SEEC  will 
give  utmost  consideration  to  leveraging  existing  initiatives  such  as  these,  where  there  is  already 
considerable  inertia  and  government/industry  support  for  improvement  activities,  before  propos¬ 
ing  new  initiatives  that  would  otherwise  be  competing  for  attention  and  resources. 

8.3  ADDITIONAL  ANALYSIS  OF  COLLECTED  DATA 

The  analysis  discussed  in  this  report  does  not  extract  all  of  the  knowledge  available  from  the  col¬ 
lected  data  set;  additional  analysis  is  possible.  Many  areas  of  study  are  possible;  two  examples  are 
presented  below. 

For  example,  responding  projects  were  executing  in  various  phases  across  the  life  cycle.  While 
data  on  Project  Performance  was  collected,  it  was  not  compared  with  position  in  the  life  cycle. 
Early  in  a  project,  estimates  of  cost-at-completion  and  project  completion  dates  seldom  vary  from 
original  estimates.  Only  as  progress  (or  lack  of  progress)  occurs  are  deviations  from  these  original 
estimates  recognized.  Thus,  on-budget  and  on-schedule  claims  later  in  a  project  are  more  credible 
than  the  same  claims  early  in  the  project.  This  factor  could  be  included  in  a  more  sophisticated 
analysis  of  Project  Performance. 

As  another  example,  the  survey  collects  data  on  organizational  CMMI  maturity  levels.  Achieve¬ 
ment  of  these  levels  requires  the  achievement  of  specified  CMMI  goals,  and  includes  the  expecta¬ 
tion  of  performance  of  various  CMMI  practices.  Many  of  the  survey  questions  assessing  SE  Ca¬ 
pabilities  are  related  to  these  same  practices.  An  analysis  of  the  consistency  between  the  claimed 
maturity  levels  and  the  performance  of  practices  could  reveal  the  degree  of  deployment  of  CMMI 
from  the  organizational  level  to  the  project  level. 

8.4  PERIODIC  REPEAT  OF  THE  SURVEY 

Broader  representation  of  programs  and  companies  across  the  defense  industrial  base  could  help 
provide  additional  insight  beyond  this  initial  survey  analysis.  As  government-  and  industry-based 
initiatives  prevail,  one  could  also  expect  to  see  improvements  in  SE  Capabilities  applied  to  pro¬ 
jects. 

Meanwhile,  defense  systems  continue  to  reach  unprecedented  levels  of  complexity  in  a  dynamic 
environment  that  is  continually  evolving  in  areas  such  as  program  mission,  emerging  technolo¬ 
gies,  development  approaches,  tools,  teaming  relationships,  and  acquisition  strategies. 
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A  periodic  re-execution  of  this  survey  and  similar  subsequent  efforts  could  quantify  the  improve¬ 
ments  in  SE  Capability,  and  could  also  ascertain  the  impact  of  these  changes  on  Project  Perform¬ 
ance. 

8.5  SURVEY  OF  ACQUIRERS 

Everything  in  this  survey  is  presented  from  the  perspective  of  the  supplier.  Additional  knowledge 
could  be  gained  by  examining  projects  from  the  perspective  of  the  acquirer.  This  could  be  accom¬ 
plished  through  the  development  of  a  similar,  but  not  identical,  questionnaire.  While  it  would  be 
valuable  to  be  able  to  join  the  results  of  the  current  survey  with  such  an  acquirer  survey,  this  most 
probably  will  not  be  feasible  due  to  the  anonymous  nature  of  the  current  survey  data.  Addressing 
both  perspectives  together  while  maintaining  confidence  in  nondisclosure  is  an  important  chal¬ 
lenge  for  future  work  in  this  area.. 
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Analysis  of  CMMI  to  Identify  and  Select  SE- 
related  Work  Products 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

Organizational  Process  Focus 

SG  1 :  Determine  Proc¬ 
ess-Improvement  Oppor¬ 
tunities  -  Strengths, 
weaknesses,  and  im- 
provement  opportunities 
for  the  organization's 
processes  are  identified 
periodically  and  as 
needed. 

SP  1.1-1:  Establish  Organizational  Process 

Needs  -  Establish  and  maintain  the  description  of 
the  process  needs  and  objectives  for  the  organi¬ 
zation. 

Organization’s  process  needs  and  objectives 

SP  1 .2-1 :  Appraise  the  Organization’s  Processes 
-  Appraise  the  processes  of  the  organization  peri- 
odically  and  as  needed  to  maintain  an  under¬ 
standing  of  their  strengths  and  weaknesses. 

Plans  for  the  organization's  process  appraisals 

Appraisal  findings  that  address  strengths  and  weaknesses  of  the  organi¬ 
zation's  processes 

Improvement  recommendations  for  the  organization's  processes 

SP  1.3-1:  Identify  the  Organization's  Process 
Improvements  -  Identify  improvements  to  the 
organization's  processes  and  process  assets. 

Analysis  of  candidate  process  improvements 

Identification  of  improvements  for  the  organization's  processes 

SG  2:  Plan  and  Imple¬ 
ment  Process- 
Improvement  Activities  - 
Improvements  are 
planned  and  imple¬ 
mented,  organizational 
process  assets  are  de¬ 
ployed,  and  process- 
related  experiences  are 
incorporated  into  the 
organizational  process 
assets. 

SP  2.1-1:  Establish  Process  Action  Plans  -  Estab¬ 
lish  and  maintain  process  action  plans  to  address 
improvements  to  the  organization's  processes 
and  process  assets. 

Organization's  approved  process  action  plans 

SP  2.2-1:  Implement  Process  Action  Plans  -  Im¬ 
plement  process  action  plans  across  the  organi- 
zation. 

Commitments  among  the  various  process  action  teams 

Status  and  results  of  implementing  process  action  plans 

Plans  for  pilots 

SP  2.3-1:  Deploy  Organizational  Process  Assets  - 
Deploy  organizational  process  assets  across  the 
organization. 

Plans  for  deploying  the  organizational  process  assets  and  changes  to 
organizational  process  assets 

Training  materials  for  deploying  the  organizational  process  assets  and 
changes  to  organizational  process  assets 

Documentation  of  changes  to  the  organizational  process  assets 

Support  materials  for  deploying  the  organizational  process  assets  and 
changes  to  organizational  process  assets 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

SP  2.4-1:  Incorporate  Process-Related  Experi¬ 
ences  into  the  Organizational  Process  Assets  - 
Incorporate  process-related  work  products, 
measures,  and  improvement  information  derived 
from  planning  and  performing  the  process  into  the 
organizational  process  assets. 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 

GP  5.1:  Ensure  Continuous  Process  Improvement 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

Optimizing  Process 

GP  5.2:  Correct  Root  Causes  of  Problem 

Organizational  Process  Definition 

SG  1 :  Establish  Organ¬ 
izational  Process  Assets 
-  A  set  of  organizational 
process  assets  is  estab¬ 
lished  and  maintained. 

SP  1.1-1:  Establish  Standard  Processes  -  Estab¬ 
lish  and  maintain  the  organization's  set  of  stan¬ 
dard  processes. 

Organization's  set  of  standard  processes 

Y 

Y 

PD01 

SP  1.2-1:  Establish  Life-Cycle  Model  Descriptions 
-  Establish  and  maintain  descriptions  of  the  life¬ 
cycle  models  approved  for  use  in  the  organiza¬ 
tion. 

Descriptions  of  life-cycle  models 

SP  1.3-1:  Establish  Tailoring  Criteria  and  Guide¬ 
lines  -  Establish  and  maintain  the  tailoring  criteria 
and  guidelines  for  the  organization's  set  of  stan¬ 
dard  processes. 

Tailoring  guidelines  for  the  organization's  set  of  standard  processes 

SP  1.4-1:  Establish  the  Organization’s  Measure¬ 
ment  Repository  -  Establish  and  maintain  the 
organization’s  measurement  repository. 

Definition  of  the  common  set  of  product  and  process  measures  for  the 
organization's  set  of  standard  processes 

Design  of  the  organization’s  measurement  repository 

Organization's  measurement  repository  (i.e.,  the  repository  structure  and 
support  environment) 

Organization’s  measurement  data 

SP  1.5-1:  Establish  the  Organization’s  Process 
Asset  Library  -  Establish  and  maintain  the  organi- 
zation's  process  asset  library. 

Design  of  the  organization’s  process  asset  library 

Organization's  process  asset  library 

Selected  items  to  be  included  in  the  organization’s  process  asset  library 

Catalog  of  items  in  the  organization’s  process  asset  library 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 

GP  2.1:  Establish  an  Organizational  Policy 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

Managed  Process 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Organizational  Training 

SG  1 :  Establish  an  Or¬ 
ganizational  Training 
Capability  -  A  training 
capability  that  supports 
the  organization's  man- 

SP  1.1-1:  Establish  the  Strategic  Training  Needs - 
Establish  and  maintain  the  strategic  training 
needs  of  the  organization. 

Training  needs 

Assessment  analysis 

SP  1.2-1:  Determine  Which  Training  Needs  Are 

Common  project  and  support  group  training  needs 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

agement  and  technical 
roles  is  established  and 
maintained. 

the  Responsibility  of  the  Organization  -  Determine 
which  training  needs  are  the  responsibility  of  the 
organization  and  which  will  be  left  to  the  individual 
project  or  support  group. 

Training  commitments 

SP  1.3-1:  Establish  an  Organizational  Training 
Tactical  Plan  -  Establish  and  maintain  an  organ¬ 
izational  training  tactical  plan. 

Organizational  training  tactical  plan 

SP  1 .4-1 :  Establish  Training  Capability  -  Establish 
and  maintain  training  capability  to  address  organ¬ 
izational  training  needs. 

Training  materials  and  supporting  artifacts 

SG  2:  Provide  Necessary 
Training  -  Training  nec¬ 
essary  for  individuals  to 
perform  their  roles  effec¬ 
tively  is  provided. 

SP  2.1-1:  Deliver  Training  -  Deliver  the  training 
following  the  organizational  training  tactical  plan. 

Delivered  training  course 

SP  2.2-1:  Establish  Training  Records  -  Establish 
and  maintain  records  of  the  organizational  train- 
ing. 

Training  records 

Training  updates  to  the  organizational  repository 

SP  2.3-1:  Assess  Training  Effectiveness  -  Assess 
the  effectiveness  of  the  organization’s  training 
program. 

Training-effectiveness  surveys 

Training  program  performance  assessments 

Instructor  evaluation  forms 

Training  examinations 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Organizational  Process  Performance 

SG  1 :  Establish  Perform¬ 
ance  Baselines  and  Mod¬ 
els  -  Baselines  and  mod¬ 
els  that  characterize  the 
expected  process  per- 
formance  of  the  organi¬ 
zation's  set  of  standard 
processes  are  estab¬ 
lished  and  maintained. 

SP  1.1-1:  Select  Processes  -  Select  the  proc¬ 
esses  or  process  elements  in  the  organization's 
set  of  standard  processes  that  are  to  be  included 
in  the  organization's  process  performance  analy¬ 
ses. 

List  of  processes  or  process  elements  identified  for  process  performance 
analyses 

SP  1.2-1:  Establish  Process  Performance  Meas¬ 
ures  -  Establish  and  maintain  definitions  of  the 
measures  that  are  to  be  included  in  the  organiza¬ 
tion's  process  performance  analyses. 

Definitions  for  the  selected  measures  of  process  performance 

SP  1.3-1:  Establish  Quality  and  Process- 
Performance  Objectives  -  Establish  and  maintain 
quantitative  objectives  for  quality  and  process 
performance  for  the  organization. 

Organization's  quality  and  process-performance  objectives 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

SP  1.4-1:  Establish  Process  Performance  Base¬ 
lines  -  Establish  and  maintain  the  organization's 
process  performance  baselines. 

Baseline  data  on  the  organization’s  process  performance 

SP  1.5-1:  Establish  Process  Performance  Models 
-  Establish  and  maintain  the  process  performance 
models  for  the  organization's  set  of  standard 
processes. 

Process  performance  models 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Organizational  Innovation  and  Deployment 

SG  1 :  Select  Improve¬ 
ments  -  Process  and 
technology  improve- 
ments  that  contribute  to 
meeting  quality  and 
process-performance 
objectives  are  selected. 

SP  1.1-1:  Collect  and  Analyze  Improvement  Pro¬ 
posals  -  Collect  and  analyze  process-  and  tech¬ 
nology-improvement  proposals. 

Analyzed  process-  and  technology-improvement  proposals 

SP  1.2-1:  Identify  and  Analyze  Innovations  -  Iden¬ 
tify  and  analyze  innovative  improvements  that 
could  increase  the  organization’s  quality  and 
process  performance. 

Candidate  innovative  improvements 

Analysis  of  proposed  innovative  improvements 

SP  1.3-1:  Pilot  Improvements  -  Pilot  process  and 
technology  improvements  to  select  which  ones  to 
implement. 

Pilot  evaluation  reports 

Documented  lessons  learned  from  pilots 

SP  1 .4-1 :  Select  Improvements  for  Deployment  - 
Select  process-  and  technology-improvement 
proposals  for  deployment  across  the  organization. 

Process-  and  technology-improvement  proposals  selected  for  deploy¬ 
ment 

SG  2:  Deploy  Improve¬ 
ments  -  Measurable 
improvements  to  the 
organization's  processes 
and  technologies  are 
continually  and  system¬ 
atically  deployed. 

SP  2.1-1:  Plan  the  Deployment  -  Establish  and 
maintain  the  plans  for  deploying  the  selected 
process  and  technology  improvements. 

Deployment  plan  for  selected  process  and  technology  improvements 

SP  2.2-1:  Manage  the  Deployment  -  Manage  the 
deployment  of  the  selected  process  and  technol¬ 
ogy  improvements. 

Updated  training  materials  (to  reflect  deployed  process  and  technology 
improvements) 

Documented  results  of  process-  and  technology-improvement  deploy¬ 
ment  activities 

Revised  process-  and  technology-improvement  measures,  objectives, 
priorities,  and  deployment  plans 

SP  2.3-1:  Measure  Improvement  Effects  -  Meas¬ 
ure  the  effects  of  the  deployed  process  and  tech¬ 
nology  improvements. 

Documented  measures  of  the  effects  resulting  from  the  deployed  proc¬ 
ess  and  technology  improvements 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Project  Planning 

SG  1 :  Establish  Esti- 

SP  1.1-1:  Estimate  the  Scope  of  the  Project- 

Task  descriptions 

Y 

Y 

PD02a 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

mates  -  Estimates  of 
project  planning  parame¬ 
ters  are  established  and 
maintained. 

Establish  a  top-level  work  breakdown  structure 
(WBS)  to  estimate  the  scope  of  the  project. 

Work  package  descriptions 

Y 

Y 

PD02a 

WBS 

Y 

Y 

PD02b 

SP  1.2-1:  Establish  Estimates  of  Work  Product 
and  Task  Attributes  -  Establish  and  maintain  es- 
timates  of  the  attributes  of  the  work  products  and 
tasks. 

Technical  approach 

Y 

Y 

PD03a 

Size  and  complexity  of  tasks  and  work  products 

Y 

Estimating  models 

Y 

Attribute  estimates 

Y 

SP  1.3-1:  Define  Project  Life  Cycle  -  Define  the 
project  life-cycle  phases  upon  which  to  scope  the 
planning  effort. 

Project  life-cycle  phases 

Y 

SP  1 .4-1 :  Determine  Estimates  of  Effort  and  Cost 
-  Estimate  the  project  effort  and  cost  for  the  work 
products  and  tasks  based  on  estimation  rationale. 

Estimation  rationale 

Y 

Project  effort  estimates 

Y 

Project  cost  estimates 

Y 

SG  2:  Develop  a  Project 
Plan  -  A  project  plan  is 
established  and  main¬ 
tained  as  the  basis  for 
managing  the  project. 

SP  2.1-1:  Establish  the  Budget  and  Schedule  - 
Establish  and  maintain  the  project’s  budget  and 
schedule. 

Project  schedules 

Y 

Schedule  dependencies 

Y 

Project  budget 

Y 

SP  2.2-1:  Identify  Project  Risks  -  Identify  and 
analyze  project  risks. 

Identified  risks 

Y 

Risk  impacts  and  probability  of  occurrence 

Y 

Risk  priorities 

Y 

SP  2.3-1 :  Plan  for  Data  Management  -  Plan  for 
the  management  of  project  data. 

Data  management  plan 

Master  list  of  managed  data 

Data  content  and  format  description 

Data  requirements  lists  for  acquirers  and  for  suppliers 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

Privacy  requirements 

Security  requirements 

Security  procedures 

Mechanism  for  data  retrieval,  reproduction,  and  distribution 

Schedule  for  collection  of  project  data 

Listing  of  project  data  to  be  collected 

SP  2.4-1 :  Plan  for  Project  Resources  -  Plan  for 
necessary  resources  to  perform  the  project. 

WBS  work  packages 

WBS  task  dictionary 

Staffing  requirements  based  on  project  size  and  scope 

Critical  facilities/equipment  list 

Process/workflow  definitions  and  diagrams 

Program  administration  requirements  list 

SP  2.5-1:  Plan  for  Needed  Knowledge  and  Skills  - 
Plan  for  knowledge  and  skills  needed  to  perform 
the  project. 

Inventory  of  skill  needs 

Staffing  and  new  hire  plans 

Databases  (e.g.,  skills  and  training) 

SP  2.6-1:  Plan  Stakeholder  Involvement  -  Plan 
the  involvement  of  identified  stakeholders. 

Stakeholder  involvement  plan 

SP  2.7-1:  Establish  the  Project  Plan  -  Establish 
and  maintain  the  overall  project  plan  content. 

Overall  project  plan 

Integrated  Master  Plan 

Y 

Y 

PD04 

Integrated  Master  Schedule 

Y 

Y 

PD05a 

Systems  Engineering  Management  Plan 

Y 

Y 

PD05c 

Systems  Engineering  Master  Schedule 

Y 

Y 

PD05b 

Systems  Engineering  Detailed  Schedule 

Y 

Y 

PD05a 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

SG  3:  Obtain  Commit¬ 
ment  to  the  Plan  -  Com¬ 
mitments  to  the  project 
plan  are  established  and 
maintained. 

SP  3.1-1 :  Review  Plans  that  Affect  the  Project  - 
Review  all  plans  that  affect  the  project  to  under¬ 
stand  project  commitments. 

Record  of  the  reviews  of  plans  that  affect  the  project 

SP  3.2-1:  Reconcile  Work  and  Resource  Levels  - 
Reconcile  the  project  plan  to  reflect  available  and 
estimated  resources. 

Revised  methods  and  corresponding  estimating  parameters  (e.g.,  better 
tools,  use  of  off-the-shelf  components) 

Renegotiated  budgets 

Revised  schedules 

Revised  requirements  list 

Renegotiated  stakeholder  agreements 

SP  3.3-1 :  Obtain  Plan  Commitment  -  Obtain 
commitment  from  relevant  stakeholders  responsi- 
ble  for  performing  and  supporting  plan  execution. 

Documented  requests  for  commitments 

Documented  commitments 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

stakeholders  include  SE  staff> 

Y 

Y 

PD02c, 

PD03b 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man- 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Project  Monitoring  and  Control 

SG  1 :  Monitor  Project 
Against  Plan  -  Actual 
performance  and  pro¬ 
gress  of  the  project  are 
monitored  against  the 
project  plan. 

SP  1.1-1:  Monitor  Project  Planning  Parameters  - 
Monitor  the  actual  values  of  the  project  planning 
parameters  against  the  project  plan. 

Records  of  project  performance 

Y 

Y 

PerfOl 
Perf02b 
Perf02e 
Cont09 
Conti  3 

Records  of  significant  deviations 

Y 

Y 

Perf02d 

SP  1 .2-1 :  Monitor  Commitments  -  Monitor  com¬ 
mitments  against  those  identified  in  the  project 
plan. 

Records  of  commitment  reviews 

SP  1 .3-1 :  Monitor  Project  Risks  -  Monitor  risks 
against  those  identified  in  the  project  plan. 

Records  of  project  risk  monitoring 

Y 

Y 

PDIIc 

SP  1 .4-1 :  Monitor  Data  Management  -  Monitor  the 
management  of  project  data  against  the  project 
plan. 

Records  of  data  management 

SP  1 .5-1 :  Monitor  Stakeholder  Involvement  - 
Monitor  stakeholder  involvement  against  the  pro¬ 
ject  plan. 

Records  of  stakeholder  involvement 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

SP  1.6-1:  Conduct  Progress  Reviews  -  Periodi¬ 
cally  review  the  project's  progress,  performance, 
and  issues. 

Documented  project  review  results 

Y 

Y 

RD12 

SP  1.7-1:  Conduct  Milestone  Reviews  -  Review 
the  accomplishments  and  results  of  the  project  at 
selected  project  milestones. 

Documented  milestone  review  results 

Y 

Y 

V&V03 

SG  2:  Manage  Corrective 
Action  to  Closure  -  Cor¬ 
rective  actions  are  man¬ 
aged  to  closure  when  the 
project's  performance  or 
results  deviate  signifi¬ 
cantly  from  the  plan. 

SP  2.1-1:  Analyze  Issues  -  Collect  and  analyze 
the  issues  and  determine  the  corrective  actions 
necessary  to  address  the  issues. 

List  of  issues  needing  corrective  actions 

Y 

Y 

Operf05 

Operf06 

V&V02d 

SP  2.2-1 :  Take  Corrective  Action  -  Take  correc¬ 
tive  action  on  identified  issues. 

Corrective  action  plan 

Y 

SP  2.3-1 :  Manage  Corrective  Action  -  Manage 
corrective  actions  to  closure. 

Corrective  action  results 

Y 

Y 

V&V02d 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Supplier  Agreement  Management 

SG  1:  Establish  Supplier 
Agreements  -  Agree¬ 
ments  with  the  suppliers 
are  established  and 
maintained. 

SP  1.1-1:  Determine  Acquisition  Type  -  Determine 
the  type  of  acquisition  for  each  product  or  product 
component  to  be  acquired. 

List  of  the  acquisition  types  that  will  be  used  for  all  products  and  product 
components  to  be  acquired 

SP  1 .2-1 :  Select  Suppliers  -  Select  suppliers 
based  on  an  evaluation  of  their  ability  to  meet  the 
specified  requirements  and  established  criteria. 

List  of  candidate  suppliers 

Preferred  supplier  list 

Rationale  for  selection  of  suppliers 

Advantages  and  disadvantages  of  candidate  suppliers 

Evaluation  criteria 

Solicitation  materials  and  requirements 

SP  1.3-1:  Establish  Supplier  Agreements  -  Estab¬ 
lish  and  maintain  formal  agreements  with  the 
supplier. 

Statements  of  work 

Contracts 

Memoranda  of  agreement 

Licensing  agreement 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

SG  2:  Satisfy  Supplier 
Agreements  -  Agree¬ 
ments  with  the  suppliers 
are  satisfied  by  both  the 
project  and  the  supplier. 

SP  2.1-1:  Review  COTS  Products  -  Review  can- 
didate  COTS  products  to  ensure  they  satisfy  the 
specified  requirements  that  are  covered  under  a 
supplier  agreement. 

Trade  studies 

Y 

Price  lists 

Y 

Evaluation  criteria 

Y 

Supplier  performance  reports 

Y 

Reviews  of  COTS  products 

Y 

SP  2.2-1 :  Execute  the  Supplier  Agreement  -  Per¬ 
form  activities  with  the  supplier  as  specified  in  the 
supplier  agreement. 

Supplier  progress  reports  and  performance  measures 

Supplier  review  materials  and  reports 

Action  items  tracked  to  closure 

Documentation  of  product  and  document  deliveries 

SP  2.3-1 :  Accept  the  Acquired  Product  -  Ensure 
that  the  supplier  agreement  is  satisfied  before 
accepting  the  acquired  product. 

Acceptance  test  procedures 

Acceptance  test  results 

Discrepancy  reports  or  corrective  action  plans 

SP  2.4-1:  Transition  Products  -  Transition  the 
acquired  products  from  the  supplier  to  the  project. 

Transition  plans 

Training  reports 

Support  and  maintenance  reports 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Integrated  Project  Management  for  IPPD 

SG  1 :  Use  the  Project’s 
Defined  Process  -  The 
project  is  conducted 
using  a  defined  process 
that  is  tailored  from  the 
organization's  set  of 
standard  processes. 

SP  1.1-1:  Establish  the  Project’s  Defined  Process 
-  Establish  and  maintain  the  project's  defined 
process. 

The  project’s  defined  process 

SP  1 .2-1 :  Use  Organizational  Process  Assets  for 
Planning  Project  Activities  -  Use  the  organiza- 
tional  process  assets  and  measurement  reposi¬ 
tory  for  estimating  and  planning  the  project’s  ac¬ 
tivities. 

Project  estimates 

Project  plans 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

SP  1.3-1:  Integrate  Plans  -  Integrate  the  project 
plan  and  the  other  plans  that  affect  the  project  to 
describe  the  project’s  defined  process. 

Integrated  plans 

SP  1.4-1:  Manage  the  Project  Using  the  Inte¬ 
grated  Plans  -  Manage  the  project  using  the  pro¬ 
ject  plan,  the  other  plans  that  affect  the  project, 
and  the  project’s  defined  process. 

Work  products  created  by  performing  the  project’s  defined  process 

Collected  measures  (“actuals”)  and  progress  records  or  reports 

Revised  requirements,  plans,  and  commitments 

Integrated  plans 

SP  1.5-1:  Contribute  to  the  Organizational  Proc¬ 
ess  Assets  -  Contribute  work  products,  measures, 
and  documented  experiences  to  the  organiza- 
tional  process  assets. 

Proposed  improvements  to  the  organizational  process  assets 

Actual  process  and  product  measures  collected  from  the  project 

Documentation  (e.g.,  exemplary  process  descriptions,  plans,  training 
modules,  checklists,  and  lessons  learned) 

SG  2:  Coordinate  and 
Collaborate  with  Rele¬ 
vant  Stakeholders  -  Co¬ 
ordination  and  collabora¬ 
tion  of  the  project  with 
relevant  stakeholders  is 
conducted. 

SP  2.1-1:  Manage  Stakeholder  Involvement - 
Manage  the  involvement  of  the  relevant  stake- 
holders  in  the  project. 

Agendas  and  schedules  for  collaborative  activities 

Documented  issues  (e.g.,  issues  with  customer  requirements,  product 
and  product-component  requirements,  product  architecture,  and  product 
design) 

Recommendations  for  resolving  relevant  stakeholder  issues 

SP  2.2-1:  Manage  Dependencies  -  Participate 
with  relevant  stakeholders  to  identify,  negotiate, 
and  track  critical  dependencies. 

Defects,  issues,  and  action  items  resulting  from  reviews  with  relevant 
stakeholders 

Critical  dependencies 

Commitments  to  address  critical  dependencies 

Status  of  critical  dependencies 

SP  2.3-1:  Resolve  Coordination  Issues  -  Resolve 
issues  with  relevant  stakeholders. 

Relevant  stakeholder  coordination  issues 

Status  of  relevant  stakeholder  coordination  issues 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

SG  3:  Use  the  Project's 
Shared  Vision  for  IPPD  - 
The  project  is  conducted 
using  the  project’s 
shared  vision. 

SP  3.1-1:  Define  Project’s  Shared-Vision  Context 
-  Identify  expectations,  constraints,  interfaces, 
and  operational  conditions  applicable  to  the  pro- 
ject’s  shared  vision. 

Organizational  expectations  and  constraints  that  apply  to  the  project 

Summary  of  project  members'  personal  aspirations  for  the  project 

External  interfaces  that  the  project  is  required  to  observe 

Operational  conditions  that  affect  the  project’s  activities 

Project’s  shared-vision  context 

SP  3.2-1:  Establish  the  Project’s  Shared  Vision  - 
Establish  and  maintain  a  shared  vision  for  the 
project. 

Meeting  minutes  for  team-building  exercises 

Shared  vision  and  objective  statements 

Statement  of  values  and  principles 

Communications  strategy 

Handbook  for  new  members  of  the  project 

Presentations  to  relevant  stakeholders 

Published  principles,  shared-vision  statement,  mission  statement,  and 
objectives  (e.g.,  posters,  wallet  cards  published  on  posters  suitable  for 
wall  hanging) 

SG  4:  Organize  Inte¬ 
grated  Teams  for  IPPD  - 
The  integrated  teams 
needed  to  execute  the 
project  are  identified, 
defined,  structured,  and 
tasked. 

SP  4.1-1 :  Determine  Integrated  Team  Structure 
for  the  Project  -  Determine  the  integrated  team 
structure  that  will  best  meet  the  project  objectives 
and  constraints. 

Assessments  of  the  product  and  product  architectures,  including  risk  and 
complexity 

Y 

Integrated  team  structures  based  on  the  WBS  and  adaptations 

Y 

Y 

Proj05 

Proj06 

Alternative  concepts  for  integrated  team  structures  that  include  respon¬ 
sibilities,  scope,  and  interfaces 

Y 

Selected  integrated  team  structure 

Y 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

SP  4.2-1:  Develop  a  Preliminary  Distribution  of 
Requirements  to  Integrated  Teams  -  Develop  a 
preliminary  distribution  of  requirements,  responsi- 
bilities,  authorities,  tasks,  and  interfaces  to  teams 
in  the  selected  integrated  team  structure. 

Preliminary  distribution  of  integrated  team  authorities  and  responsibilities 

Y 

Preliminary  distribution  of  the  work  product  requirements,  technical  inter¬ 
faces,  and  business  (e.g.,  cost  accounting,  project  management)  inter¬ 
faces  each  integrated  team  will  be  responsible  for  satisfying 

Y 

SP  4.3-1:  Establish  Integrated  Teams  -  Establish 
and  maintain  teams  in  the  integrated  team  struc- 
ture. 

A  list  of  project  integrated  teams 

Y 

List  of  team  leaders 

Y 

Responsibilities  and  authorities  for  each  integrated  team 

Y 

Y 

Proj03 

Requirements  allocated  to  each  integrated  team 

Y 

Measures  for  evaluating  the  performance  of  integrated  teams 

Y 

Y 

Proj04 

Quality  assurance  reports 

Y 

Periodic  status  reports 

Y 

New  integrated  team  structures 

Y 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Risk  Management 

SG  1 :  Prepare  for  Risk 
Management  -  Prepara¬ 
tion  for  risk  management 
is  conducted. 

SP  1.1-1:  Determine  Risk  Sources  and  Catego¬ 
ries  -  Determine  risk  sources  and  categories. 

Risk  source  lists  (external  and  internal) 

Risk  categories  list 

SP  1.2-1:  Define  Risk  Parameters  -  Define  the 
parameters  used  to  analyze  and  categorize  risks, 
and  the  parameters  used  to  control  the  risk  man¬ 
agement  effort. 

Risk  evaluation,  categorization,  and  prioritization  criteria 

Risk  management  requirements  (control  and  approval  levels,  reassess¬ 
ment  intervals,  etc.) 

SP  1.3-1:  Establish  a  Risk  Management  Strategy 
-  Establish  and  maintain  the  strategy  to  be  used 
for  risk  management. 

Project  risk  management  strategy 

SG  2:  Identify  and  Ana¬ 
lyze  Risks  -  Risks  are 

SP  2.1-1:  Identify  Risks  -  Identify  and  document 
the  risks. 

List  of  identified  risks,  including  the  context,  conditions,  and  conse¬ 
quences  of  risk  occurrence 

Y 

Y 

PDIIa 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

identified  and  analyzed 
to  determine  their  relative 
importance. 

SP  2.2-1:  Evaluate,  Categorize,  and  Prioritize 

Risks  -  Evaluate  and  categorize  each  identified 
risk  using  the  defined  risk  categories  and  parame¬ 
ters,  and  determine  its  relative  priority. 

List  of  risks,  with  a  priority  assigned  to  each  risk 

Y 

Y 

SG  3:  Mitigate  Risks  - 
Risks  are  handled  and 
mitigated,  where  appro¬ 
priate,  to  reduce  adverse 
impacts  on  achieving 
objectives. 

SP  3.1-1:  Develop  Risk  Mitigation  Plans  -  De¬ 
velop  a  risk  mitigation  plan  for  the  most  important 
risks  to  the  project,  as  defined  by  the  risk  man- 
agement  strategy. 

Documented  handling  options  for  each  identified  risk 

Y 

Y 

Risk  mitigation  plans 

Y 

Y 

PDIIb 

Contingency  plans 

Y 

Y 

PDIIb 

List  of  those  responsible  for  tracking  and  addressing  each  risk 

Y 

Y 

SP  3.2-1:  Implement  Risk  Mitigation  Plans  -  Moni- 
tor  the  status  of  each  risk  periodically  and  imple- 
ment  the  risk  mitigation  plan  as  appropriate. 

Updated  lists  of  risk  status 

Updated  assessments  of  risk  likelihood,  consequence,  and  thresholds 

Updated  lists  of  risk-handling  options 

Updated  list  of  actions  taken  to  handle  risks 

Risk  mitigation  plans 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 
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PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Integrated  Teaming 

SG  1 :  Establish  Team 
Composition  -  A  team 
composition  that  pro¬ 
vides  the  knowledge  and 
skills  required  to  deliver 
the  team’s  product  is 
established  and  main¬ 
tained. 

SP  1.1-1:  Identify  Team  Tasks  -  Identify  and  de¬ 
fine  the  team’s  specific  internal  tasks  to  generate 
the  team’s  expected  output. 

Descriptions  of  internal  work  tasks 

List  of  results  the  team  is  expected  to  achieve  for  all  work  tasks 

SP  1 .2-1 :  Identify  Needed  Knowledge  and  Skills  - 
Identify  the  knowledge,  skills,  and  functional  ex- 
pertise  needed  to  perform  team  tasks. 

List  of  disciplines  or  functions  required  to  perform  the  tasks 

List  of  the  knowledge,  key  skills,  and  critical  expertise 

Initial  profiles  of  team  skills  and  knowledge  for  the  core  team  and  the 
extended  team 

SP  1.3-1:  Assign  Appropriate  Team  Members  - 
Assign  the  appropriate  personnel  to  be  team 
members  based  on  required  knowledge  and 
skills. 

Set  of  selection  criteria 

Revised  skills  matrix  and  knowledge  profiles 

List  of  team  members 
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PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

List  of  the  level  of  effort  and  resources,  including  access  to  staff,  to  per¬ 
form  each  team  function 

SG  2:  Govern  Team 
Operation  -  Operation  of 
the  integrated  team  is 
governed  according  to 
established  principles. 

SP  2.1-1:  Establish  a  Shared  Vision  -  Establish 
and  maintain  a  shared  vision  for  the  integrated 
team  that  is  aligned  with  any  overarching  or  high- 
er  level  vision. 

Boundary  conditions  and  interfaces  within  which  the  team  must  operate 

Documented  shared  vision 

Presentation  materials  of  the  shared-vision  statement  suitable  for  team 
members  and  various  audiences  that  need  to  be  informed 

SP  2.2-1:  Establish  a  Team  Charter  -  Establish 
and  maintain  a  team  charter  based  on  the  inte- 
grated  team’s  shared  vision  and  overall  team 
objectives. 

Team  charter 

Procedures  for  setting  the  expectations  for  the  work  to  be  done  and  for 
measuring  team  performance 

List  of  critical  success  factors 

List  of  specific  strategies  the  team  expects  to  employ 

SP  2.3-1:  Define  Roles  and  Responsibilities  - 
Clearly  define  and  maintain  each  team  member’s 
roles  and  responsibilities. 

Descriptions  of  roles  and  responsibilities 

Assignment  statements 

Responsibility  matrix 

SP  2.4-1:  Establish  Operating  Procedures  -  Es- 
tablish  and  maintain  integrated  team  operating 
procedures. 

Operating  procedures  and  ground  rules 

Procedures  for  work  expectations  and  performance  measures 

SP  2.5-1:  Collaborate  among  Interfacing  Teams  - 
Establish  and  maintain  collaboration  among  inter- 
facing  teams. 

Work  product  and  process  deployment  charts 

Input  to  the  integrated  master  plan  and  integrated  schedules 

Team  work  plans 

Commitment  lists 
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SE  WP 

Key  SE  WP 

Survey 

Question 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Integrated  Supplier  Management 

SG  1 :  Analyze  and  Se- 

SP  1.1-1:  Analyze  Potential  Sources  of  Products  - 

List  of  potential  sources  of  products  that  might  be  acquired 
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SE  WP 

Key  SE  WP 

Survey 

Question 

lect  Sources  of  Products 
-  Potential  sources  of 
products  that  best  fit  the 
needs  of  the  project  are 
identified,  analyzed,  and 
selected. 

Identify  and  analyze  potential  sources  of  products 
that  may  be  used  to  satisfy  the  project’s  require- 
ments. 

Market  studies 

Trade  studies 

Information  about  potential  sources  such  as  past  performance,  post¬ 
delivery  support,  corporate  viability,  and  risks 

SP  1.2-1:  Evaluate  and  Determine  Sources  of 
Products  -  Use  a  formal  evaluation  process  to 
determine  which  sources  of  custom-made  and  off- 
the-shelf  products  to  use. 

Analysis  and  evaluation  reports 

Revised  list  of  product  sources 

SG  2:  Coordinate  Work 
with  Suppliers  -  Work  is 
coordinated  with  suppli¬ 
ers  to  ensure  the  supplier 
agreement  is  executed 
appropriately. 

SP  2.1-1:  Monitor  Selected  Supplier  Processes  - 
Monitor  and  analyze  selected  processes  used  by 
the  supplier. 

List  of  processes  selected  for  monitoring 

Activity  reports 

Performance  reports 

Performance  curves 

Discrepancy  reports 

SP  2.2-1:  Evaluate  Selected  Supplier  Work  Prod- 
ucts  -  For  custom-made  products,  evaluate  se- 
lected  supplier  work  products. 

List  of  work  products  selected  for  monitoring 

Activity  reports 

Discrepancy  reports 

SP  2.3-1 :  Revise  the  Supplier  Agreement  or  Rela¬ 
tionship  -  Revise  the  supplier  agreement  or  rela- 
tionship,  as  appropriate,  to  reflect  changes  in 
conditions. 

Revisions  to  the  supplier  agreement 

Revisions  to  the  project’s  and  supplier’s  processes  and  work  products 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

NATIONAL  DEFENSE  INDUSTRIAL  ASSOCIATION 


SOFTWARE  ENGINEERING  INSTITUTE  |  133 


Goal 

PRACTICE 
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SE  WP 

Key  SE  WP 

Survey 

Question 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Quantitative  Project  Management 

SG  1 :  Quantitatively 
Manage  the  Project  -  The 
project  is  quantitatively 
managed  using  quality 
and  process- 
performance  objectives. 

SP  1.1-1:  Establish  the  Project’s  Objectives  - 
Establish  and  maintain  the  project’s  quality  and 
process-performance  objectives. 

The  project’s  quality  and  process-performance  objectives 

SP  1.2-1:  Compose  the  Defined  Process  -  Select 
the  subprocesses  that  compose  the  project’s 
defined  process  based  on  historical  stability  and 
capability  data. 

Criteria  used  in  identifying  which  subprocesses  are  valid  candidates  for 
inclusion  in  the  project’s  defined  process 

Candidate  subprocesses  for  inclusion  in  the  project’s  defined  process 
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Survey 

Question 

Subprocesses  to  be  included  in  the  project’s  defined  process 

Identified  risks  when  selected  subprocesses  lack  a  process  performance 
history 

SP  1 .3-1 :  Select  the  Subprocesses  that  Will  Be 
Statistically  Managed  -  Select  the  subprocesses 
of  the  project's  defined  process  that  will  be  statis- 
tically  managed. 

Quality  and  process-performance  objectives  that  will  be  addressed  by 
statistical  management 

Criteria  used  in  selecting  which  subprocesses  will  be  statistically  man¬ 
aged 

Subprocesses  that  will  be  statistically  managed 

Identified  process  and  product  attributes  of  the  selected  subprocesses 
that  should  be  measured  and  controlled 

SP  1.4-1:  Manage  Project  Performance  -  Monitor 
the  project  to  determine  whether  the  project’s 
objectives  for  quality  and  process  performance 
will  be  satisfied,  and  identify  corrective  action  as 
appropriate. 

Estimates  (predictions)  of  the  achievement  of  the  project’s  quality  and 
process-performance  objectives 

Documentation  of  the  risks  in  achieving  the  project’s  quality  and  proc¬ 
ess-performance  objectives 

Documentation  of  actions  needed  to  address  the  deficiencies  in  achiev¬ 
ing  the  project’s  objectives 

SG  2:  Statistically  Man¬ 
age  Subprocess  Per¬ 
formance  -  The  perform¬ 
ance  of  selected 
subprocesses  within  the 
project's  defined  process 
is  statistically  managed. 

SP  2.1-1:  Select  Measures  and  Analytic  Tech¬ 
niques  -  Select  the  measures  and  analytic  tech- 
niques  to  be  used  in  statistically  managing  the 
selected  subprocesses. 

Definitions  of  the  measures  and  analytic  techniques  to  be  used  in  (or 
proposed  for)  statistically  managing  the  subprocesses 

Operational  definitions  of  the  measures,  their  collection  points  in  the 
subprocesses,  and  how  the  integrity  of  the  measures  will  be  determined 

Traceability  of  measures  back  to  the  project’s  quality  and  process- 
performance  objectives 

Instrumented  organizational  support  environment  to  support  automatic 
data  collection 

SP  2.2-1:  Apply  Statistical  Methods  to  Understand 

Collected  measures 
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PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

Variation  -  Establish  and  maintain  an  understand¬ 
ing  of  the  variation  of  the  selected  subprocesses 
using  the  selected  measures  and  analytic  tech- 
niques. 

Natural  bounds  of  process  performance  for  each  measured  attribute  of 
each  selected  subprocess 

Process  performance  compared  to  the  natural  bounds  of  process  per¬ 
formance  for  each  measured  attribute  of  each  selected  subprocess 

SP  2.3-1 :  Monitor  Performance  of  the  Selected 
Subprocesses  -  Monitor  the  performance  of  the 
selected  subprocesses  to  determine  their  capabil- 
ity  to  satisfy  their  quality  and  process- 
performance  objectives,  and  identify  corrective 
action  as  necessary. 

Natural  bounds  of  process  performance  for  each  selected  subprocess 
compared  to  its  established  (derived)  objectives 

For  each  subprocess,  its  process  capability 

For  each  subprocess,  the  actions  needed  to  address  deficiencies  in  its 
process  capability 

SP  2.4-1:  Record  Statistical  Management  Data  - 
Record  statistical  and  quality  management  data  in 
the  organization’s  measurement  repository. 

Statistical  and  quality  management  data  recorded  in  the  organization’s 
measurement  repository 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 
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SE  WP 

Key  SE  WP 

Survey 

Question 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Requirements  Management 

SG  1 :  Manage  Require¬ 
ments  -  Requirements 
are  managed  and  incon¬ 
sistencies  with  project 
plans  and  work  products 
are  identified. 

SP  1.1-1:  Obtain  an  Understanding  of  Require¬ 
ments  -  Develop  an  understanding  with  the  re¬ 
quirements  providers  on  the  meaning  of  the  re- 
quirements. 

Lists  of  criteria  for  distinguishing  appropriate  requirements  providers 

Y 

Y 

RD04 

Criteria  for  evaluation  and  acceptance  of  requirements 

Y 

Y 

RD05 

Results  of  analyses  against  criteria 

Y 

Y 

An  agreed-to  set  of  requirements 

Y 

Y 

RD06 

SP  1.2-2:  Obtain  Commitment  to  Requirements  - 
Obtain  commitment  to  the  requirements  from  the 
project  participants. 

Requirements  impact  assessments 

Y 

Y 

RD07 

Documented  commitments  to  requirements  and  requirements  changes 

Y 

Y 

SP  1.3-1:  Manage  Requirements  Changes  - 
Manage  changes  to  the  requirements  as  they 
evolve  during  the  project. 

Requirements  status 

Y 

Y 

Requirements  database 

Y 

Requirements  decision  database 

Y 

SP  1.4-2:  Maintain  Bidirectional  Traceability  of 
Requirements  -  Maintain  bidirectional  traceability 
among  the  requirements  and  the  project  plans 

Requirements  traceability  matrix 

Y 

Y 

Requirements  tracking  system 

Y 

Y 

RD09 
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PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

and  work  products. 

SP  1.5-1:  Identify  Inconsistencies  between  Pro¬ 
ject  Work  and  Requirements  -  Identify  inconsis- 
tencies  between  the  project  plans  and  work  prod¬ 
ucts  and  the  requirements. 

Documentation  of  inconsistencies  including  sources,  conditions,  and 
rationale 

Y 

Corrective  actions 

Y 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

Y 

Y 

RDlOa 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 

GP  5.1:  Ensure  Continuous  Process  Improvement 
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SE  WP 

Key  SE  WP 

Survey 

Question 

Optimizing  Process 

GP  5.2:  Correct  Root  Causes  of  Problem 

Requirements  Development 

SG  1 :  Develop  Customer 
Requirements  -  Stake¬ 
holder  needs,  expecta¬ 
tions,  constraints,  and 
interfaces  are  collected 
and  translated  into  cus¬ 
tomer  requirements. 

SP  1.1-1:  Collect  Stakeholder  Needs  -  Identify 
and  collect  stakeholder  needs,  expectations, 
constraints,  and  interfaces  for  all  phases  of  the 
product  life  cycle. 

SP  1 .1-2:  Elicit  Needs  -  Elicit  stakeholder  needs, 
expectations,  constraints,  and  interfaces  for  all 
phases  of  the  product  life  cycle. 

SP  1.2-1:  Develop  the  Customer  Requirements  - 
Transform  stakeholder  needs,  expectations,  con- 
straints,  and  interfaces  into  customer  require- 
ments. 

Customer  requirements 

Y 

Y 

RDOIa 

Customer  constraints  on  the  conduct  of  verification 

Y 

Customer  constraints  on  the  conduct  of  validation 

Y 

SG  2:  Develop  Product 
Requirements  -  Cus¬ 
tomer  requirements  are 
refined  and  elaborated  to 
develop  product  and 
product-component  re¬ 
quirements. 

SP  2.1-1:  Establish  Product  and  Product- 
Component  Requirements  -  Establish  and  main- 
tain  product  and  product-component  require- 
ments,  which  are  based  on  the  customer  re¬ 
quirements. 

Derived  requirements 

Y 

Y 

RDOlb 

Product  requirements 

Y 

Y 

Product-component  requirements 

Y 

SP  2.2-1:  Allocate  Product-Component  Require- 
ments  -  Allocate  the  requirements  for  each  prod- 
uct  component. 

Requirement  allocation  sheets 

Y 

Y 

RD02 

Provisional  requirement  allocations 

Y 

Design  constraints 

Y 

Derived  requirements 

Y 

Y 

Relationships  among  derived  requirements 

Y 

SP  2.3-1:  Identify  Interface  Requirements  -  Iden¬ 
tify  interface  requirements. 

Interface  requirements 

Y 

Y 

SG  3:  Analyze  and  Vali- 

SP  3.1-1:  Establish  Operational  Concepts  and 

Operational  concept 

Y 

Y 

RD03a 
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Key  SE  WP 

Survey 

Question 

date  Requirements  -  The 
requirements  are  ana¬ 
lyzed  and  validated,  and 
a  definition  of  required 
functionality  is  devel¬ 
oped. 

Scenarios  -  Establish  and  maintain  operational 
concepts  and  associated  scenarios. 

Product  installation,  operational,  maintenance,  and  support  concepts 

Y 

Y 

RD03c 

Disposal  concepts 

Y 

Use  cases 

Y 

Y 

RD03b 

Timeline  scenarios 

Y 

New  requirements 

Y 

SP  3.2-1:  Establish  a  Definition  of  Required  Func- 
tionality  -  Establish  and  maintain  a  definition  of 
required  functionality. 

Functional  architecture 

Y 

Activity  diagrams  and  use  cases 

Y 

Object-oriented  analysis  with  services  identified 

Y 

SP  3.3-1:  Analyze  Requirements  -  Analyze  re- 
quirements  to  ensure  that  they  are  necessary  and 
sufficient. 

Requirements  defects  reports 

Y 

Proposed  requirements  changes  to  resolve  defects 

Y 

Key  requirements 

Y 

Technical  performance  measures 

Y 

SP  3.4-3:  Analyze  Requirements  to  Achieve  Bal¬ 
ance  -  Analyze  requirements  to  balance  stake¬ 
holder  needs  and  constraints. 

Assessment  of  risks  related  to  requirements 

Y 

SP  3.5-1:  Validate  Requirements  -  Validate  re¬ 
quirements  to  ensure  the  resulting  product  will 
perform  appropriately  in  its  intended-use  envi¬ 
ronment. 

Results  of  requirements  validation 

Y 

SP  3.5-2:  Validate  Requirements  with  Compre¬ 
hensive  Methods  -  Validate  requirements  to  en¬ 
sure  the  resulting  product  will  perform  as  intended 
in  the  user's  environment  using  multiple  tech¬ 
niques  as  appropriate. 

Record  of  analysis  methods  and  results 

Y 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Technical  Solution 

SG  1 :  Select  Product- 

SP  1 .1-1:  Develop  Alternative  Solutions  and  Se- 

Alternative  solutions 

Y 

Y 

RD12 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

Component  Solutions  - 
Product  or  product- 
component  solutions  are 
selected  from  alternative 
solutions. 

lection  Criteria  -  Develop  alternative  solutions  and 
selection  criteria. 

Selection  criteria 

Y 

Y 

RD12 

SP  1.1-2:  Develop  Detailed  Alternative  Solutions 
and  Selection  Criteria  -  Develop  detailed  alterna- 
tive  solutions  and  selection  criteria. 

Alternative  solution  screening  criteria 

Y 

Y 

Evaluations  of  new  technologies 

Y 

Alternative  solutions 

Y 

Y 

Selection  criteria  for  final  selection 

Y 

Y 

SP  1.2-2:  Evolve  Operational  Concepts  and  Sce¬ 
narios  -  Evolve  the  operational  concept,  scenar¬ 
ios,  and  environments  to  describe  the  conditions, 
operating  modes,  and  operating  states  specific  to 
each  product  component. 

Product-component  operational  concepts,  scenarios,  and  environments 
for  all  product-related  life-cycle  processes  (e.g.,  operations,  support, 
training,  manufacturing,  deployment,  fielding,  delivery,  and  disposal) 

Y 

Timeline  analyses  of  product-component  interactions 

Y 

Use  cases 

Y 

SP  1.3-1:  Select  Product-Component  Solutions  - 
Select  the  product-component  solutions  that  best 
satisfy  the  criteria  established. 

Product-component  selection  decisions  and  rationale 

Y 

Y 

Documented  relationships  between  requirements  and  product  compo¬ 
nents 

Y 

Documented  solutions,  evaluations,  and  rationale 

Y 

SG  2:  Develop  the  De¬ 
sign  -  Product  or  product- 
component  designs  are 
developed. 

SP  2.1-1:  Design  the  Product  or  Product  Compo¬ 
nent  -  Develop  a  design  for  the  product  or  product 
component. 

Product  architecture 

Y 

Y 

IF03a, 

IF03b 

Product-component  designs 

Y 

SP  2.2-3:  Establish  a  Technical  Data  Package 

Technical  data  package 

Y 

SP  2.3-1:  Establish  Interface  Descriptions  -  Es¬ 
tablish  and  maintain  the  solution  for  product- 
component  interfaces. 

Interface  design 

Y 

Y 

IF01 

Interface  design  documents 

Y 

Y 

IF02 

SP  2.3-3:  Design  Interfaces  Using  Criteria  -  De¬ 
sign  comprehensive  product-component  inter- 
faces  in  terms  of  established  and  maintained 

Interface  design  specifications 

Y 

Interface  control  documents 

Y 

Y 

IF02 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

criteria. 

Interface  specification  criteria 

Y 

Rationale  for  selected  interface  design 

Y 

SP  2.4-3:  Perform  Make,  Buy,  or  Reuse  Analyses 
-  Evaluate  whether  the  product  components 
should  be  developed,  purchased,  or  reused 
based  on  established  criteria. 

Criteria  for  design  and  product-component  reuse 

Y 

Make-or-buy  analyses 

Y 

Y 

Guidelines  for  choosing  COTS  product  components 

Y 

Y 

IF04 

SG  3:  Implement  the 
Product  Design  -  Product 
components,  and  asso¬ 
ciated  support  documen¬ 
tation,  are  implemented 
from  their  designs. 

SP  3.1-1:  Implement  the  Design  -  Implement  the 
designs  of  the  product  components. 

Implemented  design 

Y 

Standard  Parts  Lists 

Y 

Standard  drawing  requirements 

Y 

International  Organization  for  Standardization  (ISO)  T3303  standards  for 
manufactured  parts 

Y 

SP  3.2-1:  Develop  Product  Support  Documenta¬ 
tion  -  Develop  and  maintain  the  end-use  docu- 
mentation. 

End-user  training  materials 

Y 

User's  manual 

Y 

Operator's  manual 

Y 

Maintenance  manual 

Y 

Online  help 

Y 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

Y 

Y 

RD11 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Product  Integration 

SG  1 :  Prepare  for  Prod¬ 
uct  Integration  -  Prepara¬ 
tion  for  product  integra- 
tion  is  conducted. 

SP  1.1-1:  Determine  Integration  Sequence  -  De¬ 
termine  the  product-component  integration  se- 
quence. 

Product  integration  sequence 

Y 

Rationale  for  selecting  or  rejecting  integration  sequences 

Y 

SP  1.2-2:  Establish  the  Product  Integration  Envi¬ 
ronment  -  Establish  and  maintain  the  environment 
needed  to  support  the  integration  of  the  product 
components. 

Verified  environment  for  product  integration 

Y 

Support  documentation  for  the  product  integration  environment 

Y 

SP  1.3-3:  Establish  Product  Integration  Proce¬ 
dures  and  Criteria  -  Establish  and  maintain  pro- 
cedures  and  criteria  for  integration  of  the  product 
components. 

Product  integration  procedures 

Y 

Y 

IF05 

Product  integration  criteria 

Y 

SG  2:  Ensure  Interface 

SP  2.1-1:  Review  Interface  Descriptions  for  Com- 

Categories  of  interfaces 

Y 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

Compatibility  -  The  prod¬ 
uct-component  inter¬ 
faces,  both  internal  and 
are  compatible. 

pleteness  -  Review  interface  descriptions  for  cov- 
erage  and  completeness. 

List  of  interfaces  per  category 

Y 

Mapping  of  the  interfaces  to  the  product  components  and  product  inte¬ 
gration  environment 

Y 

SP  2.2-1:  Manage  Interfaces  -  Manage  internal 
and  external  interface  definitions,  designs,  and 
changes  for  products  and  product  components. 

Table  of  relationships  among  the  product  components  and  the  external 
environment  (e.g.,  main  power  supply,  fastening  product,  computer  bus 
system) 

Y 

Table  of  relationships  between  the  different  product  components 

Y 

List  of  agreed-to  interfaces  defined  for  each  pair  of  product  components, 
when  applicable 

Y 

Reports  from  the  interface  control  working  group  meetings 

Y 

Action  items  for  updating  interfaces 

Y 

Application  program  interface  (API) 

Y 

Updated  interface  description  or  agreement 

Y 

SG  3:  Assemble  Product 
Components  and  Deliver 
the  Product  -  Verified 
product  components  are 
assembled  and  the  inte¬ 
grated,  verified,  and 
validated  product  is  de¬ 
livered. 

SP  3.1-1:  Confirm  Readiness  of  Product  Compo- 
nents  for  Integration  -  Confirm,  prior  to  assembly, 
that  each  product  component  required  to  assem- 
ble  the  product  has  been  properly  identified,  func¬ 
tions  according  to  its  description,  and  that  the 
product-component  interfaces  comply  with  the 
interface  descriptions. 

Acceptance  documents  for  the  received  product  components 

Delivery  receipts 

Checked  packing  lists 

Exception  reports 

Waivers 

SP  3.2-1 :  Assemble  Product  Components  -  As¬ 
semble  product  components  according  to  the 
product  integration  sequence  and  available  pro¬ 
cedures. 

Assembled  product  or  product  components 

SP  3.3-1:  Evaluate  Assembled  Product  Compo- 

Exception  reports 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

nents  -  Evaluate  assembled  product  components 
for  interface  compatibility. 

Interface  evaluation  reports 

Product  integration  summary  reports 

SP  3.4-1:  Package  and  Deliver  the  Product  or 
Product  Component  -  Package  the  assembled 
product  or  product  component  and  deliver  it  to  the 
appropriate  customer. 

Packaged  product  or  product  components 

Delivery  documentation 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Verification 

SG  1 :  Prepare  for  Verifi¬ 
cation  -  Preparation  for 
verification  is  conducted. 

SP  1.1-1:  Select  Work  Products  for  Verification  - 
Select  the  work  products  to  be  verified  and  the 
verification  methods  that  will  be  used  for  each. 

Lists  of  work  products  selected  for  verification 

Y 

Verification  methods  for  each  selected  work  product 

Y 

SP  1.2-2:  Establish  the  Verification  Environment  - 
Establish  and  maintain  the  environment  needed 
to  support  verification. 

Verification  environment 

Y 

SP  1 .3-3:  Establish  Verification  Procedures  and 
Criteria  -  Establish  and  maintain  verification  pro- 
cedures  and  criteria  for  the  selected  work  prod¬ 
ucts. 

Verification  procedures 

Y 

Y 

V&VOIa 

Verification  criteria 

Y 

Y 

V&VOlb 

SG  2:  Perform  Peer 
Reviews  -  Peer  reviews 
are  performed  on  se- 

SP  2.1-1 :  Prepare  for  Peer  Reviews  -  Prepare  for 
peer  reviews  of  selected  work  products. 

Peer  review  schedule 

Y 

Peer  review  checklist 

Y 

lected  work  products. 

Entry  and  exit  criteria  for  work  products 

Y 

Y 

V&V02a 

Criteria  for  requiring  another  peer  review 

Y 

Peer  review  training  material 

Y 

Y 

V&V02b 

Selected  work  products  to  be  reviewed 

Y 

Y 

V&V02C 

SP  2.2-1 :  Conduct  Peer  Reviews  -  Conduct  peer 
reviews  on  selected  work  products  and  identify 
issues  resulting  from  the  peer  review. 

Peer  review  results 

Y 

Y 

Peer  review  issues 

Y 

Y 

V&V02e 

Peer  review  data 

Y 

SP  2.3-2:  Analyze  Peer  Review  Data  -  Analyze 

Peer  review  data 

Y 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

data  about  preparation,  conduct,  and  results  of 
the  peer  reviews. 

Peer  review  action  items 

Y 

Y 

V&V02d 

SG  3:  Verify  Selected  Work  Products  -  Selected 
work  products  are  verified  against  their  specified 
requirements. 

SP  3.1-1:  Perform  Verification  -  Perform  verifica¬ 
tion  on  the  selected  work  products. 

Verification  results 

Verification  reports 

Demonstrations 

As-run  procedures  log 

SP  3.2-2:  Analyze  Verification  Results  and  Iden¬ 
tify  Corrective  Action  -  Analyze  the  results  of  all 
verification  activities  and  identify  corrective  action. 

Analysis  report  (such  as  statistics  on  performances,  causal  analysis  of 
nonconformances,  comparison  of  the  behavior  between  the  real  product 
and  models,  and  trends) 

Y 

Trouble  reports 

Y 

Y 

Change  requests  for  the  verification  methods,  criteria,  and  environment 

Y 

Y 

Corrective  actions  to  verification  methods,  criteria,  and/or  environment 

Y 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Validation 

SG  1 :  Prepare  for  Valida¬ 
tion  -  Preparation  for 
validation  is  conducted. 

SP  1.1-1:  Select  Products  for  Validation  -  Select 
products  and  product  components  to  be  validated 
and  the  validation  methods  that  will  be  used  for 
each. 

Lists  of  products  and  product  components  selected  for  validation 

Y 

Validation  methods  for  each  product  or  product  component 

Y 

Requirements  for  performing  validation  for  each  product  or  product  com¬ 
ponent 

Y 

Validation  constraints  for  each  product  or  product  component 

Y 

SP  1.2-2:  Establish  the  Validation  Environment  - 
Establish  and  maintain  the  environment  needed 
to  support  validation. 

Validation  environment 

Y 

SP  1.3-3:  Establish  Validation  Procedures  and 

Validation  procedures 

Y 

Y 

V&V04a 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

Criteria  -  Establish  and  maintain  procedures  and 
criteria  for  validation. 

Validation  criteria 

Y 

Y 

V&\ /04b 

Test  and  evaluation  procedures  for  maintenance,  training,  and  support 

Y 

SG  2:  Validate  Product 
or  Product  Components  - 
The  product  or  product 
components  are  vali¬ 
dated  to  ensure  that  they 
are  suitable  for  use  in 
their  intended  operating 
environment. 

SP  2.1-1:  Perform  Validation  -  Perform  validation 
on  the  selected  products  and  product  compo- 
nents. 

Validation  reports 

Validation  results 

Validation  cross-reference  matrix 

As-run  procedures  log 

Operational  demonstrations 

SP  2.2-1:  Analyze  Validation  Results  -  Analyze 
the  results  of  the  validation  activities  and  identify 
issues. 

Validation  deficiency  reports 

Y 

Validation  issues 

Y 

Procedure  change  request 

Y 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Configuration  Management 

SG  1 :  Establish  Base¬ 
lines  -  Baselines  of  iden¬ 
tified  work  products  are 
established. 

SP  1.1-1:  Identify  Configuration  Items  -  Identify 
the  configuration  items,  components,  and  related 
work  products  that  will  be  placed  under  configura¬ 
tion  management. 

Identified  configuration  items 

Y 

Y 

V&V05 

SP  1.2-1:  Establish  a  Configuration  Management 
System  -  Establish  and  maintain  a  configuration 
management  and  change  management  system 
for  controlling  work  products. 

Configuration  management  system  with  controlled  work  products 

Y 

Configuration  management  system  access  control  procedures 

Y 

Y 

Change  request  database 

Y 

Y 

SP  1.3-1:  Create  or  Release  Baselines  -  Create 
or  release  baselines  for  internal  use  and  for  deliv- 
ery  to  the  customer. 

Baselines 

Y 

Y 

V&V08 

P  erf 01 

Description  of  baselines 

Y 

SG  2:  Track  and  Control 
Changes  -  Changes  to 

SP  2.1-1:  Track  Change  Requests  -  Track 
change  requests  for  the  configuration  items. 

Change  requests 

Y 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

the  work  products  under 
configuration  manage¬ 
ment  are  tracked  and 
controlled. 

SP  2.2-1:  Control  Configuration  Items  -  Control 
changes  to  the  configuration  items. 

Revision  history  of  configuration  items 

Y 

Y 

V&V06 

Archives  of  the  baselines 

Y 

Y 

V&V08 

SG  3:  Establish  Integrity 
-  Integrity  of  baselines  is 
established  and  main¬ 
tained. 

SP  3.1-1:  Establish  Configuration  Management 
Records  -  Establish  and  maintain  records  de- 
scribing  configuration  items. 

Revision  history  of  configuration  items 

Y 

Y 

V&V07 

Change  log 

Y 

Copy  of  the  change  requests 

Y 

Status  of  configuration  items 

Y 

Differences  between  baselines 

Y 

SP  3.2-1 :  Perform  Configuration  Audits  -  Perform 
configuration  audits  to  maintain  integrity  of  the 
configuration  baselines. 

Configuration  audit  results 

Y 

Action  items 

Y 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Process  and  Product  Quality  Assurance 

SG  1 :  Objectively  Evalu¬ 
ate  Processes  and  Work 
Products  -  Adherence  of 
the  performed  process 
and  associated  work 
products  and  services  to 
applicable  process  de¬ 
scriptions,  standards, 
and  procedures  is  objec¬ 
tively  evaluated. 

SP  1.1-1:  Objectively  Evaluate  Processes  -  Ob- 
jectively  evaluate  the  designated  performed  proc¬ 
esses  against  the  applicable  process  descrip- 
tions,  standards,  and  procedures. 

Evaluation  reports 

Noncompliance  reports 

Corrective  actions 

SP  1 .2-1 :  Objectively  Evaluate  Work  Products 
and  Services  -  Objectively  evaluate  the  desig- 
nated  work  products  and  services  against  the 
applicable  process  descriptions,  standards,  and 
procedures. 

Evaluation  reports 

Noncompliance  reports 

Corrective  actions 

SG  2:  Provide  Objective 
Insight  -  Noncompliance 
issues  are  objectively 
tracked  and  communi¬ 
cated,  and  resolution  is 
ensured. 

SP  2.1-1:  Communicate  and  Ensure  Resolution  of 
Noncompliance  Issues  -  Communicate  quality 
issues  and  ensure  resolution  of  noncompliance 
issues  with  the  staff  and  managers. 

Corrective  action  reports 

Evaluation  reports 

Quality  trends 

SP  2.2-1:  Establish  Records  -  Establish  and  main¬ 
tain  records  of  the  quality  assurance  activities. 

Evaluation  logs 

Quality  assurance  reports 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

Status  reports  of  corrective  actions 

Reports  of  quality  trends 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

Measurement  and  Analysis 

SG  1 :  Align  Measure¬ 
ment  and  Analysis  Activi¬ 
ties  -  Measurement  ob¬ 
jectives  and  activities  are 
aligned  with  identified 
information  needs  and 
objectives. 

SP  1.1-1:  Establish  Measurement  Objectives  - 
Establish  and  maintain  measurement  objectives 
that  are  derived  from  identified  information  needs 
and  objectives. 

Measurement  objectives 

SP  1 .2-1 :  Specify  Measures  -  Specify  measures 
to  address  the  measurement  objectives. 

Specifications  of  base  and  derived  measures 

SP  1.3-1:  Specify  Data  Collection  and  Storage 
Procedures  -  Specify  how  measurement  data  will 
be  obtained  and  stored. 

Data  collection  and  storage  procedures 

Data  collection  tools 

SP  1 .4-1 :  Specify  Analysis  Procedures  -  Specify 
how  measurement  data  will  be  analyzed  and 
reported. 

Analysis  specification  and  procedures 

Data  analysis  tools 

SG  2:  Provide  Measure¬ 
ment  Results  -  Meas¬ 
urement  results  that 
address  identified  infor¬ 
mation  needs  and  objec¬ 
tives  are  provided. 

SP  2.1-1:  Collect  Measurement  Data  -  Obtain 
specified  measurement  data. 

Base  and  derived  measurement  data  sets 

Results  of  data  integrity  tests 

SP  2.2-1:  Analyze  Measurement  Data  -  Analyze 
and  interpret  measurement  data. 

Analysis  results  and  draft  reports 

SP  2.3-1 :  Store  Data  and  Results  -  Manage  and 
store  measurement  data,  measurement  specifica¬ 
tions,  and  analysis  results. 

Stored  data  inventory 

SP  2.4-1:  Communicate  Results  -  Report  results 
of  measurement  and  analysis  activities  to  all  rele- 
vant  stakeholders. 

Delivered  reports  and  related  analysis  results 

Contextual  information  or  guidance  to  aid  in  the  interpretation  of  analysis 
results 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Decision  Analysis  and  Resolution 

SG  1 :  Evaluate  Alterna¬ 
tives  -  Decisions  are 
based  on  an  evaluation 
of  alternatives  using 
established  criteria. 

SP  1.1-1:  Establish  Guidelines  for  Decision  Anal¬ 
ysis  -  Establish  and  maintain  guidelines  to  deter¬ 
mine  which  issues  are  subject  to  a  formal  evalua¬ 
tion  process. 

Guidelines  for  when  to  apply  a  formal  evaluation  process 

Y 

SP  1.2-1:  Establish  Evaluation  Criteria  -  Establish 
and  maintain  the  criteria  for  evaluating  alterna- 
tives,  and  the  relative  ranking  of  these  criteria. 

Documented  evaluation  criteria 

Y 

Rankings  of  criteria  importance 

Y 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

SP  1.3-1:  Identify  Alternative  Solutions  -  Identify 
alternative  solutions  to  address  issues. 

Identified  alternatives 

Y 

SP  1 .4-1 :  Select  Evaluation  Methods  -  Select  the 
evaluation  methods. 

Selected  evaluation  methods 

Y 

SP  1.5-1:  Evaluate  Alternatives  -  Evaluate  alter¬ 
native  solutions  using  the  established  criteria  and 
methods. 

Evaluation  results 

Y 

SP  1 .6-1 :  Select  Solutions  -  Select  solutions  from 
the  alternatives  based  on  the  evaluation  criteria. 

Recommended  solutions  to  address  significant  issues 

Y 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 

Organizational  Environment  for  Integration 

SG  1:  Provide  IPPD 
Infrastructure  -  An  infra¬ 
structure  that  maximizes 
the  productivity  of  people 
and  affects  the  collabora¬ 
tion  necessary  for  inte- 
gration  is  provided. 

SP  1.1-1:  Establish  the  Organization’s  Shared 
Vision  -  Establish  and  maintain  a  shared  vision  for 
the  organization. 

Organization’s  shared  vision 

Evaluations  of  the  organization’s  shared  vision 

Guidelines  for  shared-vision  building  within  projects  and  integrated 
teams 

SP  1.2-1:  Establish  an  Integrated  Work  Environ¬ 
ment  -  Establish  and  maintain  an  integrated  work 
environment  that  supports  IPPD  by  enabling  col- 
laboration  and  concurrent  development. 

Requirements  for  the  integrated  work  environment 

Design  of  the  integrated  work  environment 

Integrated  work  environment 

SP  1.3-1:  Identify  IPPD-Unique  Skill  Require¬ 
ments  -  Identify  the  unique  skills  needed  to  sup- 
port  the  IPPD  environment. 

IPPD  strategic  training  needs 

IPPD  tactical  training  needs 

SG  2:  Manage  People  for 
Integration  -  People  are 
managed  to  nurture  the 
integrative  and  collabora¬ 
tive  behaviors  of  an  IPPD 
environment. 

SP  2.1-1:  Establish  Leadership  Mechanisms  - 
Establish  and  maintain  leadership  mechanisms  to 
enable  timely  collaboration. 

Guidelines  for  determining  the  degree  of  empowerment  of  people  and 
integrated  teams 

Guidelines  for  setting  leadership  and  decision-making  context 

Organizational  process  documentation  for  issue  resolution 

SP  2.2-1:  Establish  Incentives  for  Integration  - 
Establish  and  maintain  incentives  for  adopting 
and  demonstrating  integrative  and  collaborative 
behaviors  at  all  levels  of  the  organization. 

Policies  and  procedures  for  performance  appraisal  and  recognition  that 
reinforce  collaboration 

Integrated  team  and  individual  recognition  and  rewards 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

SP  2.3-1 :  Establish  Mechanisms  to  Balance 

Team  and  Flome  Organization  Responsibilities  - 
Establish  and  maintain  organizational  guidelines 
to  balance  team  and  home  organization  responsi¬ 
bilities. 

Organizational  guidelines  for  balancing  team  and  home  organization 
responsibilities 

Performance  review  process  that  considers  both  functional  supervisor 
and  team  leader  input 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 

GP  5.1:  Ensure  Continuous  Process  Improvement 
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Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

Optimizing  Process 

GP  5.2:  Correct  Root  Causes  of  Problem 

Causal  Analysis  and  Resolution 

SG  1 :  Determine  Causes 
of  Defects  -  Root  causes 
of  defects  and  other 
problems  are  systemati¬ 
cally  determined. 

SP  1.1-1:  Select  Defect  Data  for  Analysis  -  Select 
the  defects  and  other  problems  for  analysis. 

Defect  and  problem  data  selected  for  further  analysis 

Y 

SP  1.2-1:  Analyze  Causes  -  Perform  causal  anal¬ 
ysis  of  selected  defects  and  other  problems  and 
propose  actions  to  address  them. 

Action  proposal 

Y 

SG  2:  Address  Causes  of 
Defects  -  Root  causes  of 
defects  and  other  prob- 
lems  are  systematically 
addressed  to  prevent 
their  future  occurrence. 

SP  2.1-1:  Implement  the  Action  Proposals  -  Im¬ 
plement  the  selected  action  proposals  that  were 
developed  in  causal  analysis. 

Action  proposals  selected  for  implementation 

Y 

Improvement  proposals 

Y 

SP  2.2-1 :  Evaluate  the  Effect  of  Changes  -  Evalu¬ 
ate  the  effect  of  changes  on  process  perform¬ 
ance. 

Measures  of  performance  and  performance  change 

Y 

SP  2.3-1:  Record  Data  -  Record  causal  analysis 
and  resolution  data  for  use  across  the  project  and 
organization. 

Causal  analysis  and  resolution  records 

Y 

GG  1 :  Achieve  Specific 
Goals 

GP  1.1:  Perform  Base  Practices 

GG  2:  Institutionalize  a 
Managed  Process 

GP  2.1:  Establish  an  Organizational  Policy 

GP  2.2:  Plan  the  Process 

GP  2.3:  Provide  Resources 

GP  2.4:  Assign  Responsibility 

GP  2.5:  Train  People 

GP  2.6:  Manage  Configurations 

GP  2.7:  Identify  and  Involve  Relevant  Stake¬ 
holders 
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NDIA 


Goal 

PRACTICE 

WORK  PRODUCT  (WP) 

SE  WP 

Key  SE  WP 

Survey 

Question 

GP  2.8:  Monitor  and  Control  the  Process 

GP  2.9:  Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Man¬ 
agement 

GG  3:  Institutionalize  a 
Defined  Process 

GP  3.1:  Establish  a  Defined  Process 

GP  3.2:  Collect  Improvement  Information 

GG  4:  Institutionalize  a 
Quantitatively  Managed 
Process 

GP  4.1:  Establish  Quantitative  Objectives  for  the 
Process 

GP  4.2:  Stabilize  Subprocess  Performance 

GG  5:  Institutionalize  an 
Optimizing  Process 

GP  5.1:  Ensure  Continuous  Process  Improvement 

GP  5.2:  Correct  Root  Causes  of  Problem 
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APPENDIX  B  Survey  Questionnaire  (annotated 

reproduction) 


The  Effectiveness  of  Systems  Engineering:  A  Survey 

Welcome  to  your  personalized  questionnaire  for  our  survey  on  “The  Effectiveness  of  Systems 
Engineering.”  Our  hope  is  that  your  participation  will  help  your  project  and  organization  judge  the 
effectiveness  of  their  systems  engineering  practices  relative  to  the  successes  and  challenges  re¬ 
ported  by  others  throughout  the  industry. 

Most  of  the  information  necessary  to  complete  the  questionnaire  should  be  easily  accessible  or 
familiar  to  you  or  perhaps  an  informed  designee.  It  should  take  about  30  to  45  minutes  to  com¬ 
plete  the  questionnaire.  Please  provide  your  best  estimates  if  quantitative  measurements  are  un¬ 
available. 

Please  complete  the  questionnaire  as  candidly  and  completely  as  you  possibly  can.  The  results 
will  be  useful  to  you,  us  and  others  only  to  the  extent  that  all  survey  participants  do  so.  As  always, 
information  collected  under  promise  of  non  disclosure  by  the  SEI  will  be  held  in  strict  confidence. 
No  attribution  to  individual  organizations  will  be  made.  Results  will  be  reported  in  summary  ag¬ 
gregate  form,  similar  to  the  SEI  process  maturity  profiles.  There  is  no  need  to  hide  weaknesses  or 
embellish  strengths. 

Be  sure  to  save  your  personalized  URL  if  you  have  not  already  done  so.  It  is  -- 

https://seir.sei.cmu.edu/feedback/SystemsEngineering.asp? ID=xxxxxxxx.  You  or  your  designee 
may  return  to  that  URL  and  continue  completing  the  questionnaire  at  any  time.  You  also  may  save 
your  work  at  any  time.  There  are  separate  Save  buttons  for  each  section  of  the  questionnaire. 
Please  be  sure  that  you  are  fully  finished  before  you  press  the  Submit  button  at  the  end  of  the 
questionnaire. 

A  detailed  summary  report  of  the  survey  results  will  be  prepared  by  NDIA  with  the  assistance  of 
the  Software  Engineering  Institute.  For  at  least  a  full  year,  the  report  will  be  made  available  only 
to  those  who  fully  complete  a  survey  questiomiaire.  The  report  will  provide  a  baseline  against 
which  you  can  compare  the  performance  of  your  project  and  organization.  Scroll  down  to  Authen¬ 
tication  below  for  more  information. 

Thank  you  once  again  for  your  help  with  this  important  activity.  And,  please  feel  free  to  contact 
us  at  benchmark@sei.cmu.edu  if  you  have  any  difficulty  with  the  questionnaire. 

Authentication 


The  summary  survey  report  will  be  available  via  an  authenticated  Web  site.  To  check  on  the  status 
of  the  report  from  time  to  time,  please  save  the  following  address: 
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https://seir.sei.cmu.edu/feedback/SystemsEngineeringSurveyResults.htm  When  the  report  is  com¬ 
pleted  you  will  need  your  account  name  and  the  password  you  enter  below  to  access  it. 


1.  Your  account  name  is:  <xxxxxxx>  (randomly  generated  to  protect  your  anonymity) 

2.  Password  ( Please  choose  a  unique  password  of  your  own  choosing  —  protect  your  continued 
anonymity  by  avoiding  anything  that  specifies  or  hints  at  the  identity  of  yourself  your  project, 
or  your  organization) 


You  must  enter  a  password  to  save  your  work.  You  will  need  the  password  to  complete  your 
questionnaire  in  more  than  one  session.  You  will  also  need  the  password  and  account  name  in 
order  to  access  the  detailed  survey  report. 


<User  defined  Password> 


Analysis 
Data  Code 


About  This  Project 

The  information  gathered  here  and  in  the  next  few  sections  will  be  used  by  the 
survey  analysts  to  categorize  the  participating  projects  and  organizations  in  order 
to  better  understand  the  responses  to  subsequent  questions  about  systems  engi¬ 
neering  practices  and  project  performance 


ProjOla  - 
ProjOlj 


1 .  What  phases  of  the  integrated  product  life  cycle  are  or  will  be  included  in  this 
project?  ( Please  select  as  many  as  apply) 

□  Concept  Refinement 

□  Technology  Development  and  Demonstration 

□  Development 

□  Manufacturing  /  Production 

□  Verification  /  Validation 

□  Training 

□  Deployment 

□  Operation 

□  Support 

□  Disposal 
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Analysis 
Data  Code 


Proj02a  - 
Proj02j 


Proj03 

Proj04 

Proj05 

Proj06 

Proj07a 

Proj07b 

Proj8a 


2.  What  phase  or  phases  of  the  integrated  product  life  cycle  is  this  project  pres¬ 
ently  executing?  ( Please  select  as  many  as  apply ) 

□  Concept  Refinement 

□  Technology  Development  and  Demonstration 

□  Development 

□  Manufacturing  /  Production 

□  Verification  /  Validation 

□  Training 

□  Deployment 

□  Operation 

□  Support 

□  Disposal 

Following  are  several  statements  that  have  been  used  to  characterize  various  de¬ 
velopment  projects.  Flow  well  do  the  statements  describe  this  project? 

3.  This  project  uses  integrated  product  teams  (IPTs) 

□  Yes  □  No  ( Please  continue  with  question  8  below) 

4.  This  project  makes  effective  use  of  integrated  product  teams  (IPTs) 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

5.  Both  the  supplier  and  the  acquirer  actively  participate  in  IPTs 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

6.  My  suppliers  actively  participate  in  IPTs 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

7.  This  project  has  an  IPT  with  assigned  responsibility  for  systems  engineering 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

This  project  has  Systems  Engineering  representation  on  each  IPT 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

8.  The  project  is  technically  challenging  because...  ( Please  select  one  for  each) 

...there  is  no  precedent  for  what  is  being  done 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 
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Analysis 
Data  Code 

Proj8b 

...significant  constraints  are  placed  on  the  quality  attributes  (e.g.  reliability, 
scalability,  security,  supportability,  etc.)  of  the  product 
□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

Proj8c 

...the  size  of  the  development  effort  is  large 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

Proj8d 

...the  technology  needed  for  this  project  is  not  mature 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

Proj8e 

...there  are  extensive  needs  for  interoperability  with  other  systems 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

ProjSf 

...insufficient  resources  (e.g.  people,  funding)  are  available  to  support  the  pro¬ 
ject 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

Proj8g 

...insufficient  skills  and  subject  matter  expertise  are  available  to  support  the 
project 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

Proj8h 

...for  other  reasons  (Please  describe  briefly) 

Proj09 

9.  This  project  team  has  successfully  completed  projects  similar  to  this  in  the 
past. 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

10.  The  requirements  for  this  project ...  ( Please  select  one  for  each) 

ProjlOa 

...are  well-defined 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

ProjlOb 

...have  not  changed  significantly  throughout  the  life  of  the  project  to-date 
□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

About  the  Product 
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ProdOl 


Prod02a 


Prod3a — 
Prod3j 


Which  selection  best  characterizes  your  customer?  ( Please  select  one) 

□  US  Government  (including  DoD  and  other  agencies) 

□  Prime  Contractor 

□  Subcontractor 

Who  is  acquiring  this  product?  {Please  select  one) 

□  Army 

□  Navy/Marine 

□  Air  Force 

□  NASA 

□  Fiomeland  Defense 

□  DARPA 

□  Other  Government  Agency 

□  Commercial 

J  Other  (Please  describe  briefly) 


Who  is  the  primary  end  user  (or  users)  of  this  product?  (Please  select  as  many 
as  apply) 

□  Army 

□  Navy/Marine 

□  Air  Force 

□  NASA 

□  Fiomeland  Defense 

□  DARPA 

□  Other  Government  Agency 

□  Commercial 

J  Other  ( Please  describe  briefly) 
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Prod4a 


4. 


In  the  context  of  the  ultimate  product  delivered  to  the  end  user,  where  does 
this  project  fit  in  the  following  hierarchy?  ( Please  select  one) 

□  System  of  Systems  /  Family  of  Systems 

□  System 

□  Subsystem 

□  Component  (hardware  and/or  software) 

□  Process 

□  Material 

□  Other  ( Please  describe  briefly) 


Prod5a — 
Prod5f 


5.  Where  will  this  system  resulting  from  this  project  be  used?  ( Please  select  as 
many  as  apply) 

□  Land 

□  Air 

□  Sea 

□  Undersea 

□  Space 

□  Other  ( Please  describe  briefly) 


ContOl 


Cont02 


About  the  Contract 

1 .  What  is  the  current  total  contract  value  of  this  project?  ( Please  specify  in  US 
dollars  —  numbers  only,  without  a  dollar  sign  or  commas) 

_ US  dollars  ($) 

2.  What  is  the  current  total  planned  duration  of  this  project?  ( Please  specify) 

_ Calendar  months 


Cont03 


3.  What  was  the  initial  contract  value  of  this  project?  (_ Please  specify  in  US  dol¬ 
lars  —  numbers  only,  without  a  dollar  sign  or  commas) 

_ US  dollars  ($) 


NATIONAL  DEFENSE  INDUSTRIAL  ASSOCIATION 


SOFTWARE  ENGINEERING  INSTITUTE  |  167 


Analysis 
Data  Code 


Cont04 


Cont05 


Con  tO  6 


Cont07 


Cont08 


Cont09 


What  was  the  initial  total  planned  duration  of  this  project?  {Please  specify ) 
_ Calendar  months 

How  many  contract  change  orders  have  been  received?  {Please  specify  a 
number,  approximate  if  necessary) 

_ Change  orders 

Approximately  how  many  person-years  of  effort  are  allocated  to  be  spent  on 
this  project  within  your  organization?  {Please  specify  a  number ) 

_ Person  years 

What  program  acquisition  category  (ACAT  level)  is  your  program  classified 
at?  {Please  select  one ) 

□  Don't  Know 

□  ACATIAC 

□  ACATIAM 

□  ACATIC 

□  ACAT  ID 

□  ACAT  II 

□  ACAT  III 

□  Other  ( Please  describe  briefly) 


What  percentage  of  the  total  contract  value  is  subcontracted  to  your  suppliers? 
(Please  specify  an  approximate  percentage  —  without  the  percentage  sign) 

_ 1% 

What  is  the  current  completion  status  of  this  project?  (Please  specify  an  ap- 
proximate  percentage  —  without  the  percentage  sign  —  e.g.,  60%  complete) 

_ %  Complete 
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Conti Oa  - 
Conti  Oh 


Conti  1 


Conti  2 


Conti  3 


Conti  4a 


10.  How  many  stakeholders  (including  internal  and  external)  are  involved  in  this 
project?  (Please  select  one  for  each) 


Acquirers 

System  integration  contractors 

Maintenance  contractors 

Development  co-contractors  (e.g.,  sister  develop 

ment  programs) 

Development  sub-contractors 

Oversight  contractors 

Users 

Other  (Please  describe  briefly) 

11.  What  percentage  of  the  customer  technical  requirements  were  marked  “To  Be 
Determined”  at  time  of  contract  award?  ( Please  specify  —  numbers  only, 
without  the  percentage  sign ) 

_ 1% 

12.  What  percentage  of  the  customer’s  technical  requirements  are  currently 
marked  “To  Be  Determined”?  ( Please  specify  an  approximate  percentage  — 
without  the  percentage  sign ) 

_ 1% 

13.  Do  you  separately  cost  and  track  systems  engineering  activities?  ( Please  se¬ 
lect  one) 

□  Yes 

□  No  (Please  continue  with  question  15) 

□  Don't  know  (Please  continue  with  question  15) 

14.  Approximately  what  percentage  of  non-recurring  engineering  (NRE)  does 
systems  engineering  represent?  ( Please  specify  an  approximate  percentage  — 
without  the  percentage  sign) 

_ 1% 
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Conti  4b 


Is  the  NRE  percentage  estimated,  or  is  it  a  measured  value?  ( Please 
select  one) 

□  Estimated  □  Measured 


Contl5a  - 
Contl5n 


15.  What  type  of  contract(s)  was  awarded  for  this  project?  (Please  select  as  many 
as  apply) 

□  Firm  fixed  price  —  FAR  16.202 

□  Fixed  price  with  economic  price  adjustment  —  FAR  16.203 

□  Fixed  price  with  prospective  price  redetermination  —  FAR  16.205 

□  Fixed  ceiling  with  retroactive  price  redetermination  —  FAR  16.206 

□  Firm  fixed  price,  level  of  effort  —  FAR  16.207 

□  Cost  reimbursement  —  FAR  16.302 

□  Cost  sharing  -  FAR  16.303 

□  Cost  plus  incentive  fee  —  FAR  16.304 

□  Cost  plus  fixed  fee  --  FAR  16.306 

□  Fixed  price  incentive  —  FAR  16.403 

□  Fixed  price  with  award  fees  —  FAR  16.404 

□  Cost  plus  award  fee  —  FAR  16.405 

□  Other  ( Please  describe  briefly) 


About  the  Organization 

By  "organization"  we  mean  an  administrative  structure  within  which  (possibly 
many)  projects  or  similar  work  efforts  are  organized  under  common  management 
and  policies. 


When  thinking  about  your  organization,  please  answer  for  the  unit  to  which  this 
project  reports  administratively,  e.g.,  a  site,  division  or  department,  not  for  a  lar¬ 
ger  enterprise  of  which  the  organization  to  which  you  report  may  be  a  part. 


1 .  Following  are  two  statements  that  have  been  used  to  characterize  various  de¬ 
velopment  organizations.  How  well  do  the  statements  describe  this  project's 
parent  organization?  ( Please  select  one  for  each) 
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OrgOla 


OrgOlb 


Org02 


Org03a  - 
Org03k 


This  organization  has  successfully  completed  projects  similar  to  this  one  in 
the  past 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly 

Agree 

Process  improvement  efforts  in  this  organization  have  been  directly  related  to 
systems  engineering 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly 

Agree 

To  what  extent  do  the  tailored  processes  followed  by  this  project  comply  with 
the  organization’s  standard  processes?  ( Please  select  one) 

□  Highly  compliant;  processes  closely  followed 

□  Largely  compliant;  processes  usually  followed 

□  Moderately  compliant;  processes  not  always  followed 

□  Not  compliant;  or  processes  not  followed 

What  process  improvement  activities  have  been  undertaken  on  this  project? 
{Please  select  as  many  as  apply) 

□  ISO  9000 

□  Lean 

□  Six  Sigma 

□  SE-CMM 

□  SW-CMM 

□  SECAM 

□  EIA-731 

□  CMMI 

□  None 

J  Other  {Please  describe  briefly) 


NATIONAL  DEFENSE  INDUSTRIAL  ASSOCIATION 


SOFTWARE  ENGINEERING  INSTITUTE  |  171 


Analysis 
Data  Code 


Org04 


4.  At  what,  if  any,  CMM  or  CMMI  Maturity  Level  has  this  project's  parent  or¬ 
ganization  most  recently  been  appraised?  ( Please  select  one) 

□  Not  appraised  ( Please  continue  with  question  7) 

□  Level  1  (Initial) 

□  Level  2  (Managed) 

□  Level  3  (Defined) 

□  Level  4  (Quantitatively  Managed) 

□  Level  5  (Optimizing) 


Org05 


Org06 


5.  When  was  the  organization's  most  recent  appraisal?  ( Please  select  one ) 

□  Within  the  past  6  months 

□  Within  the  past  year 

□  Within  the  past  2  years 

□  More  than  2  years  ago 

6.  What  model  was  used?  ( Please  select  one ) 

□  CMMI-SE/SW/IPPD/SS 

□  CMMI-SE/SW/IPPD 

□  CMMI-SE/SW 

□  CMMI-SW 


Org07 


OrgOS 


7.  Has  this  project  been  objectively  verified  to  be  implementing  processes  con¬ 
sistent  with  a  given  CMM/CMMI  maturity  level?  ( Please  select  one) 

□  Not  verified 

□  Level  1  (Initial) 

□  Level  2  (Managed) 

□  Level  3  (Defined) 

□  Level  4  (Quantitatively  Managed) 

□  Level  5  (Optimizing) 

8.  Is  anything  else  particularly  important  in  characterizing  your  project,  the  or¬ 
ganization  within  which  it  resides  or  the  system  that  you  are  developing? 

( Please  describe  here) _ 


Process  Definition,  Project  Planning  &  Risk  Management 
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PD01 

PD02a 

PD02b 


This  and  the  next  few  sections  ask  you  about  the  systems  engineering  activities 
performed  on  this  project.  Most  of  the  questions  ask  about  the  existence  and  qual¬ 
ity  of  tangible  work  products.  Note  that  the  pertinent  infonnation  often  may  be 
distributed  throughout  multiple  documents  or  other  work  products;  it  need  not 
necessarily  be  located  in  one  particular  place. 

Following  are  several  statements  about  work  products  and  activities  that  are  some¬ 
times  used  for  systems  development.  Please  use  the  following  definitions  to  de¬ 
scribe  their  use  on  this  project: 

Strongly  Disagree  The  work  product  does  not  exist  or  is  never  used  on  this 
project. 

Disagree  The  work  product  is  of  insufficient  quality  or  is  not  used  regularly  at 
appropriate  occasions  on  this  project. 

Agree  The  work  product  or  practice  is  of  good  quality  and  it  is  used  regularly 
on  this  project,  although  not  necessarily  as  often  as  it  could  be. 

Strongly  Agree  The  work  product  or  practice  is  of  exceptional  quality  and  it  is 
used  at  nearly  all  appropriate  occasions  on  this  project. 

Not  Applicable  This  work  product  or  practice  does  not  apply  to  this  project  at 
the  current  stage  of  the  project’s  life  cycle  (e.g.,  test  reports  do  not  exist  be¬ 
cause  we  are  not  yet  in  the  verification  phase  of  the  project). 

1 .  This  project  utilizes  a  documented  set  of  systems  engineering  processes  for 
the  planning  and  execution  of  the  project 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

2.  This  project  has  an  accurate  and  up-to-date  Work  Breakdown  Structure 
(WBS)  that...  (Please  select  one  for  each ) 

...  includes  task  descriptions  and  work  package  descriptions 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...  is  based  upon  the  product  structure 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 
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PD02c 

PD02d 

PD03a 

PD03b 

PD03c 

PD04a 

PD04b 

PD04c 


...  is  developed  with  the  active  participation  of  those  who  perform  the  sys¬ 
tems  engineering  activities 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...  is  developed  with  the  active  participation  of  all  relevant  stakeholders, 
e.g.,  developers,  maintainers,  testers,  inspectors,  etc. 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

This  project’s  Technical  Approach  (i.e.  a  top-level  strategy  and  methodology 
to  create  the  initial  conceptual  design  for  product  development)...  ( Please  se¬ 
lect  one  for  each ) 

...is  complete,  accurate  and  up-to-date 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...is  developed  with  the  active  participation  of  those  who  perform  the  systems 
engineering  activities 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...is  developed  with  the  active  participation  of  all  appropriate  functional 
stakeholder 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan  (IMP), 
that...  (Please  select  one  for  each) 

...is  an  event-driven  plan  (i.e.,  each  accomplishment  is  tied  to  a  key  project 
event) 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...documents  significant  accomplishments  with  pass/fail  criteria  for  both  busi¬ 
ness  and  technical  elements  of  the  project 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...is  consistent  with  the  WBS 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

This  project  has  an  integrated  event-based  schedule  that...  {Please  select  one 
for  each) 
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PD05b 

PD05c 

PD05d 

PD05e 

PD06 

PD07 

PD08 

PD09 

PD  10 


...is  structured  as  a  networked,  multi-layered  schedule  of  project  tasks  re¬ 
quired  to  complete  the  work  effort 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...contains  a  compilation  of  key  technical  accomplishments  (e.g.,  a  Systems 
Engineering  Master  Schedule) 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...references  measurable  criteria  (usually  contained  in  the  Integrated  Master 
Plan)  required  for  successful  completion  of  key  technical  accomplish¬ 
ments 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...is  consistent  with  the  WBS 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...identifies  the  critical  path  of  the  program  schedule 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

This  project  has  a  plan  or  plans  for  the  performance  of  technical  reviews  with 
defined  entry  and  exit  criteria  throughout  the  life  cycle  of  the  project 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

This  project  has  a  plan  or  plans  that  include  details  of  the  management  of  the 
integrated  technical  effort  across  the  project  (e.g.,  a  Systems  Engineering 
Management  Plan  or  a  Systems  Engineering  Plan) 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

Those  who  perform  systems  engineering  activities  actively  participate  in  the 
development  and  updates  of  the  project  planning 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

Those  who  perform  systems  engineering  activities  actively  participate  in 
tracking/reporting  of  task  progress 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

The  acquirer  has  provided  this  project  with  a  Systems  Engineering  Plan 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


NATIONAL  DEFENSE  INDUSTRIAL  ASSOCIATION 


SOFTWARE  ENGINEERING  INSTITUTE  |  175 


Analysis 
Data  Code 

PDlla 

PD  lib 

PDllc 

PDlld 

PD12 

RDOla 

RDOlb 

RD02 


1 1 .  This  project  has  a  Risk  Management  process  that...  ( Please  select  one  for 
each) 

...creates  and  maintains  an  accurate  and  up-to-date  list  of  risks  affecting  the 
project  (e.g.,  risks  to  cost,  risks  to  schedule,  risks  to  performance) 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly 
Agree 

...creates  and  maintains  up-to-date  documentation  of  risk  mitigation  plans  and 
contingency  plans  for  selected  risks 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...monitors  and  reports  the  status  of  risk  mitigation  activities  and  resources 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...assesses  risk  against  achievement  of  an  event-based  schedule 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

12.  This  project's  Risk  Management  process  is  integrated  with  program  decision¬ 
making 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

Requirements  Development,  Requirements  Management  &  Trade  Studies 

1.  This  project  maintains  an  up-to-date  and  accurate  listing  of  all  requirements... 
{Please  select  one  for  each) 

...specified  by  the  customer,  to  include  regulatory,  statutory,  and  certification 
requirements 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...derived  from  those  specified  by  the  customer 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

2.  This  project  maintains  up-to-date  and  accurate  documentation  clearly  reflect¬ 
ing  the  hierarchical  allocation  of  both  customer  and  derived  requirements  to 
each  element  (subsystem,  component,  etc.)  of  the  system  in  the  configuration 
baselines 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 
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3.  This  project  documents  and  maintains  accurate  and  up-to-date  descriptions 
of...  ( Please  select  one  for  each) 


RD03a 


...operational  concepts  and  their  associated  scenarios 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


RD03b 


...use  cases  (or  their  equivalent) 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


RD03c 

RD04 


RD05 


...product  installation,  maintenance  and  support  concepts 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

4.  This  project  has  documented  criteria  for  identifying  authorized  requirements 
providers  to  avoid  requirements  creep  and  volatility 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

5.  This  project  has  documented  criteria  (e.g.,  cost  impact,  schedule  impact,  au¬ 
thorization  of  source,  contract  scope,  requirement  quality)  for  evaluation  and 
acceptance  of  requirements 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


RD06 


6.  The  requirements  for  this  project  are  approved  in  a  formal  and  documented 
manner  by  relevant  stakeholders 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


RD07 


7.  This  project  performs  and  documents  requirements  impact  assessments  for 
proposed  requirements  changes 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


RD08 


8.  This  project  develops  and  documents  project  requirements  based  upon  stake¬ 
holder  needs,  expectations,  and  constraints 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


RD09 


9.  This  project  has  an  accurate  and  up-to-date  requirements  tracking  system 
□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

10.  For  this  project,  the  requirements  documents  are...  ( Please  select  one  for  each ) 


RDlOa 


. .  .managed  under  a  configuration  control  process 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 
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RDlOb 

RD11 

RD12 

RD13 

IF01 

IF02 

IF03a 

IF03b 

IF03c 


...accessible  to  all  relevant  project  staff 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

1 1 .  Stakeholders  impacted  by  trade  studies  are  involved  in  the  development  and 
performance  of  those  trade  studies 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

12.  This  project  performs  and  documents  trade  studies  between  alternate  solutions 
based  upon  definitive  and  documented  selection  criteria 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

13.  Documentation  of  trade  studies  is  maintained  in  a  defined  repository  and  is 
accessible  to  all  relevant  project  staff 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

Interfaces,  Product  Structure  &  Integration 

1.  This  project  maintains  accurate  and  up-to-date  descriptions  (e.g.  interface  con¬ 
trol  documents,  models,  etc.)  defining  interfaces  in  detail 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

2.  Interface  definition  descriptions  are  maintained  in  a  designated  location,  under 
configuration  management,  and  accessible  to  all  who  need  them 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

3.  For  this  project,  the  product  high-level  structure  is...  ( Please  select  one  for 
each) 

. .  .documented,  kept  up  to  date,  and  managed  under  configuration  control 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...documented  using  multiple  views  (e.g.  functional  views,  module  views, 
etc.) 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...accessible  to  all  relevant  project  personnel 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 
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IF04 


IF05 


V&VOla 


V&VOlb 


V&V02a 


V&V02b 


V&V02c 


V&  V02d 


V&V02e 


V&  V02f 


4.  This  project  has  defined  and  documented  guidelines  for  choosing  COTS  prod¬ 
uct  components 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

5.  This  project  has  accurate  and  up-to-date  documents  defining  its  product  inte¬ 
gration  process,  plans,  criteria,  etc.  throughout  the  life  cycle 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


Verification,  Validation  &  Configuration  Management 


1.  This  project  has  accurate  and  up-to-date  documents  defining...  ( Please  select 
one  for  each ) 

...the  procedures  used  for  the  test  and  verification  of  systems  and  system  ele¬ 
ments 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...acceptance  criteria  used  for  the  verification  of  systems  and  system  elements 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

2.  This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design 
reviews,  etc.)  process  that...  (Please  select  one  for  each) 

...defines  entry  and  exit  criteria  for  work  products 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

. .  .includes  training  requirements  for  the  reviewers 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...defines  criteria  for  the  selection  of  work  products  (e.g.,  requirements  docu¬ 
ments,  test  plans,  system  design  documents,  etc.)  for  review 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...tracks  action  items  to  closure 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...addresses  identified  risks  and  risk  mitigation  activities  during  reviews 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

...examines  completeness  of  configuration  baselines 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 
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V&V03 


3.  This  project  conducts  non-advocate  reviews  (e.g.  reviews  by  qualified  person¬ 
nel  with  no  connection  to  or  stake  in  the  project)  and  documents  results,  is¬ 
sues,  action  items,  risks,  and  risk  mitigations  ( Please  select  one) 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


4.  This  project  has  accurate  and  up-to-date  documents  defining...  {Please  select 
one  for  each ) 


Vc WO  4  a 


...the  procedures  used  for  the  validation  of  systems  and  system  elements 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


V&V04b 


...acceptance  criteria  used  for  the  validation  of  systems  and  system  elements 
□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


V&V05 


5.  This  project  maintains  a  listing  of  items  managed  under  configuration  control 
□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


V&V06 


6.  This  project  has  a  configuration  management  system  that  charters  a  Change 
Control  Board  to  disposition  change  requests  {Please  select  one) 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


V&V07 


7.  This  project  maintains  records  of  requested  and  implemented  changes  to  con¬ 
figuration-managed  items 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


V&V08 


PerfOl 


8.  This  project  creates  and  manages  configuration  baselines  (e.g.,  functional, 
allocated,  product)  {Please  select  one) 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

Project  Performance:  Earned  Value  Management 

1 .  This  project  creates  and  manages  cost  and  schedule  baselines 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 


2.  Following  are  five  statements  about  Earned  Value  Management  Systems 
(EVMS).  Do  you  agree  or  disagree  that  they  describe  this  project? 


Perf02a 


Your  customer  requires  that  you  supply  EVMS  data 
□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 
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Perf02b 


Perf02c 

Perf02d 

Perf02e 

Perf03 


Perf04a. 

Perf04b 


EVMS  data  are  available  to  decision  makers  in  a  timely  manner  (i.e.  cur¬ 
rent  within  2  weeks) 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

The  requirement  to  track  and  report  EVMS  data  is  levied  upon  the  pro¬ 
ject’s  suppliers 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

Variance  thresholds  for  CPI  and  SPI  variance  are  defined,  documented, 
and  used  to  determine  when  corrective  action  is  needed 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly 
Agree 

EVMS  is  linked  to  the  technical  effort  through  the  WBS  and  the  IMP/IMS 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

When  is  the  EVMS  baseline  updated?  ( Please  select  as  many  as  apply) 

□  Only  at  contract  initiation 

□  Whenever  a  contract  change  order  or  renewal  is  received 

□  Incrementally  in  rolling  wave  planning 

□  Whenever  the  project  is  reprogrammed  due  to  a  pre-determined  cost  or 
schedule  variance 

□  At  periodic  intervals 

□  Other  (Please  describe  briefly) 


What  are  the  current  estimated  cost  at  completion  and  the  current  estimated 
total  duration  for  this  project?  (Please  specify  for  each ) 

_  Cost  in  US  Dollars  (No  dollar  sign  or  commas 

please) _ 

_  Duration  in  months 
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Perf05a. 

Perf05b 


5. 


What  is  the  projected  variance  at  completion  for  the  current  contract  baseline? 
{Please  specify  for  each,  using  +  signs  for  any  overruns  and  -  signs  for  any 
underruns) 

in  US  Dollars  (No  dollar  sign  or  commas  please) 
_  Duration  in  months 


Perf06 


6.  What  is  the  current  cumulative  (or  final)  EVMS  Cost  Performance  Index 
(CPI)  for  this  project?  (Please  specify  a  number) 


P  erf 07 


7.  What  is  the  current  cumulative  (or  final)  EVMS  Schedule  Performance  Index 
(SPI)  for  this  project?  (Please  specify  a  number) 


Perf08a  - 
PerfOSj 


8.  What,  if  any,  primary  reasons  do  you  attribute  for  the  cost  and/or  schedule 
variance  on  your  program?  (Please  select  the  1-3  most  significant  factors,  if 
applicable) 

□  Not  applicable 

□  Mis-estimated 

□  Mis-understood 

□  Unfunded  scope  growth  (requirements  creep) 

□  Insufficient  resources  (staffing,  etc.) 

□  Technical  issues 

□  COTS/reuse  issues 

□  Supplier  issues 

□  Systems  engineering  activities  insufficiently  funded  by  program  office 

□  Other  reasons 


Other  Performance  Indicators 
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OPerfOla, 

OPerfOlb 


OPerf02 


OPerf03 


OPerf04 


OPerf05 


OPerf06 


What  percentage  of  available  Award  Fees  have  been  received  by  this  project? 
(Please  specify  an  approximate  percentage  for  each  —  without  the  percentage 
sign ) _ 

%  —  in  the  current  period 

_  %  —  to  date  (i.e.,  in  all  periods) 

Requirements  are  being  satisfied  and  remain  on  track  to  be  satisfied  in  the 
product  releases  as  originally  planned;  they  are  not  being  deleted  or  deferred 
to  later  releases 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

Overall,  this  project  is  performing  per  the  schedule  established  in  the  current 
IMS  approved  by  the  acquirer 

□  Strongly  Disagree  □  Disagree  □  Agree  □  Strongly  Agree 

The  schedule  of  this  project’s  critical  path,  when  compared  to  the  current  IMS 
approved  by  the  acquirer  is...  (Please  select  one ) 

□  Greater  than  6  months  late 

□  Greater  than  3  months  late 

□  Greater  than  1  month  late 

□  Within  plus  or  minus  1  month 

□  Greater  than  1  month  early 

□  Greater  than  3  months  early 

□  Greater  than  6  months  early 

Does  this  project  track  reports  of  problems  from  fielded  items?  (Please  select 
one) 

□  Yes 

□  No  (Please  continue  with  question  8) 

Does  the  project  conduct  an  engineering  assessment  of  all  field  trouble  re¬ 
ports?  (Please  select  one ) 

□  Yes 

□  No  (Please  continue  with  question  8) 
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OPerf07a  - 
OPerf07d 


OPerf08 


OPerf09 


OP  erf  10 


OP  erf  11 


7.  The  results  of  this  engineering  assessment  feed  into  . . .  (Please  select  as  many 
as  apply ) 

□  Operational  Hazard  Risk  Assessments 

□  Materiel  Readiness  Assessments 

□  System  Upgrades  Planning 

□  Other  (Please  describe  briefly) 


8.  What  performance  indicators  (beyond  cost  and  schedule)  have  been  particu¬ 
larly  useful  for  managing  your  project?  (Please  describe  here) 


9.  What  other  kinds  of  performance  related  information  would  have  been  helpful 
for  your  project  or  program,  but  was  unavailable?  (Please  describe  here) 


10.  What  indicators  do  you  use  in  your  project  or  organization  to  determine  sys- 
tems  engineering  effectiveness?  (Please  describe  here) _ 


11.  What  indicators  of  systems  engineering  effectiveness  are  regularly  reviewed 
across  projects  by  higher  level  management?  (Please  describe  here) 


In  Conclusion 
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ConcOl 


Is  there  anything  else  that  you  would  like  to  tell  us  about  your  project  or  this  sur- 
vey?  ( Please  describe  here ) _ 


Thank  you  very  much  for  your  time  and  effort! 

Before  pressing  the  Submit  button:  Please  be  sure  to  use  the  Save  button.  Then 
use  your  browser's  File  Save  As...  to  keep  a  copy  of  your  completed  questionnaire 
on  your  local  computer.  When  doing  so,  do  not  change  the  default  file  type.  You 
may  wish  to  refer  to  the  local  copy  if  you  need  to  recall  your  account  name  and 
password  when  the  summary  results  become  available. 

Please  press  the  Submit  button  only  when  you  have  completed  the  questionnaire 
fully  to  your  satisfaction. 
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ewcasibU  -Tnly  n;-,  ni|l-.-r  .mJ  ,^f;|  gliff.  ond  'J-.I!  nee  be-  seen  hi  dnj  niki  Ci'anuj-  Lial  i-f  pnr(miT>7llil 
■:nC.miejli.:er-.  III.;  17  pt1  mri  IP  hilK'  ^tdllKiKS  Of  dlltKillill  fUClfpi.  Ik  TfMA  SpftH 
Hji|:iit,vnri;  I. fj vision.  SUPC'Olta.’tJ  In-  SL-I.  will  m|HK  >i  Jepkn  Jim  Ik  c»ull>  Jic  icim  ed  .inj  ora baid 
hi  :-;|-|.  Tk;  ri.ruilir  Trill  br  puUished  only  si  qfpigia  nw*uiwt.  wd  euseipu  ini  niku  rrwullTikuiab.:- 
^Ji:lr>  Vk-  will  pni'.i.dr  j  ci.ip_L  III  tyu  as  S4M  3S  il  IS  hJIdy. 

Wt  hd  lor  'rurf  supped  nr  lliii  iuiv-:y  Bir<vl,  anl  WHir  k-lp  in  i-s  ..-SiViJ  .  HI.  Spti.  ilka.  .  *l-  c»k.  lliaJ  jshi 
(nr  ;e-:rmr  dir-Jjjnx  1 

■  I  d  j  ii  id  1  pf-niocii .ihin  your  ui^ml.._i  ixi  e.i  |liii:i-ij-3j  i-  -  ibis  -j.i  r-  ■;  ;■ 

*  DisirJtuKllK  jiuelKil  survey  -ix-u i  1  u  1  ii s  1  l>  1 1 p.-j-  pnijoci  irtim^TTa.  oulInfrzitG  j r -J  orsciua^ir^ 

ikrn  in  -pciiiiiijii 

L  I-  >  1  ia  ■■)■  *  .  Ik-  .  ...  .1  -.1  |.  . . j,  -.  .  ...  .  .  ,■  ■!.,  pjp  ^  IfsjjLij, 

■  Kopor  TO  lilil  t.r.L-  llunii:i  >H  |:is-jivls  sulkilnl,  .ml  Ik;  nimevi  TTSpilldlnj. 


e-inm^rfi  firfinm  Unvid^i 
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QurgfrC-Gciinjgheu  ihwn  Lful  rri«c  prmKt  mji.inin  will  he:  able-  w  CfHTipklt  fte  ym^H'nnun 
in  r;F'C':j  fn:fji  30  to  60  iiii-LKi.  t‘nu  -:un  set  i  tiMi-fttuaNf  riijiy  i,ir  |Jw  ([iKM.iffi*iuirc  ,ii 
hii.7'-;:'V---jir.v.  ■: m u . l-:J il’i-j t-J h ut !■-■>.; ■■r.i-,  t-. I'j i vi 7 1  : t i 1 1 ^  ru-nm  lm?-  This  Hie  wiL  Mranc  aeiivt 
un  LOJuly-aOC*. 

Thiiik  tou  in  jU'  mil-u  fin-  |i  ir  t-ip.i-ir,;  in  ihir  iirfaum  miairft.  The  .rebuilt  £u,y  h  Jh*:rj.v  iln 
ewtatkxi  of  iyitci'iD.  cnyiit-riT^  3j  «e||  e;  jniffnrTi.'tii  Kqulslilta  ben  pu-.iiLi.--.  nJ  jwr  c*n 

■i-,'.l.«:iifji'ir  ivofL 


l.iYa  irriLi'  ?  F':jt n ■  1 1  Jr. 

1_ i ■  " ' n ^ ■  1 1  *  icnjr  il.  USftF  CRj£I> 

■'  i:lJ;iil  i  CEO 

Njiitiul  ntruwc 'hdiiK'iii  ftiswiiilan 
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E  nc  kis  ii  n.1  2:  Non- 1)  isclo^ii  re  /  Pm  acy  Policy 

L  ' 1 1  iL -  .  ur''.-;-  Li  be  Jig  .poit  ■,'ftT.I  b;-  ilk-  NDIA  and  win  h.-  erectled-*  id  Mr.-  Software  iJigiriHritv 

IibIIue  i.SEI  i-A-th  :Jk-  The  SEI  Li  311  independent  rr;> ■  i ■  i_. ■  i i ■  ■  i ■  withcul  direct  Lies  lo  either  jrcpiirer.- 
or  iiippliere  II  fe  known  fee  mairl  itniig oonl idcnUall}  '>1  i  n  fa  im  alien  recebecl  from  both 
go>  emmeit  prog-rins-  ind  defene  ocniroclois 

2  The  quei-lioniEiJe  el  icih  ii>>  intbrirulion  identic  inr.  specific  E:pondenls.  pre-guns  or  organon  Lions. 
.uppMiir.  ilk-  information, 

.1  All  'liu  Will  be  ,:if'imik  'J  iIf.hi;.  Mi"ii  l;-  il  Jig  randomh  generated  IDs*  it  the  Ski  website  Since 
rei-ponses-wm  be  SLtrrniited  eksclrenicalbp.  ii  L;  in  If.- nub  pos  :  ibk  lo  trice  the:?  responses  backlo 
ilk- J  source  The  SB  w  in  mJce  no  attempt  bo  do  so,  md  win  .quickh  rteptiue  Ur  response  din  from 
the  random  I tZe=  in  an  uni  n  cable  nviiikv 

4  I  > ii.i  pT"--  kktl  fee  ilii:  str*e»  w  ill  be  used  onl»  hr  Une .  iak-d  purports-;  of  this  sir>e> 

5  The  cLiei  win  be  encrypted  Jt  Iraitiil it  a  secure  .todoei  layer  iIf.I  stoc-j  in  encry  pted  formal  Ji  3 
:eaie  local  ion  ■'■  iiluii  the  2EL 

6  Only  authorized  SEI  staff  participating  in  Ui&suhey  aclh-ity  will  fane  access  to the  nur  data.  Sikh 
access*  in  be  on  u  need  to- know  bariL-;  hr  sir^ey  management  purposes  -.Hib  Raw  cLi  ej  will  not  be 
made  m-aiLibteloany  ether  commercial  ct  ^cE-eminenul  ungihtciiion 

1  ftnapEr-erledin  rportswiii  net  ccn  lain  air-  infomtaiion  traceable  loam  pen  on  projed.  or 

ctitaniidiFiti 

3  To  lull  Irr  prctec!  the  confident  L  ■  I  i  r-  all  ie.  pond-ms  will  be  rtcticiled  -»ia  pm;.  The  :ur»ey  its  will 
octilacl  pnoiien  a  I  each  ct-g  an  Lett  ion.  The  pox  iestwiii  solicit  rerponlenls  from  within  the 
-.tganii.tktt  Neither  the  Ski  nor  I  he  Mali  oral  l>efertr  InduraiialAssociaiionilNDlAjwiii  know 
whichprogninsot- which  prcjecl.' premium  mnugen-lm-e  been  contacted  AllcomnunicaUon  wiih 
the  SB  and  NDlAw  in  be  >ia  Ihe  pro*  is 


NATIONAL  DEFENSE  INDUSTRIAL  ASSOCIATION 


SOFTWARE  ENGINEERING  INSTITUTE  |  191 


E  lidos  un?  ?:  Project  Selection  Instruct  ions 

L  Per  puiporiesofUiissiinej,  Lhe  ck-l ihl r.Hi '.'I  j  * projec l" or "propra rrf  broil'  Ew-emiillj,  ah 
s-nnlherilii^  aa  Mi;-  Ihal  is  fcrmalh  1-  IMf.’.I  ■  bn  icotlrad  .1  .pccilica  f.h l  a  nr  mo  nil  dun  of 
apreemenl  efc  1  mil  is  nundifd  h-  up  reject  or  program  number  and  employs  r.enenilb-  iceepled 
ocrl-ac  court  iii;-  technique:- inn  Ik-  cere  i±  red a  p-ojecl  For  j  more  comp rehereii'e  definition  cj  ilk- 
lenn  "projed'’see  l-jk'lomre  5  Term- aid  De flatten 

2  Be , l r..- .J  Lh j|  Lhe  infomuiion  ilia  j  cu  supph  will  be  he  Id  in  :-tri:l  confidence  free  [ir>:lo.ure  I 

!■■'■. hi-  Discloriire I  ri-  icj  Folic* 1  Onb  luihcrled  5EI  .Halt  will  k r- 1.-  access  lo  lhe  ini- errroi  p. hi 
supplied  Tbcqu.’.-iiieiruir  doesnolisolicilar^  infonruiionlo  identify  ill-  person  projector 
ceiiintu  ion  re  ponding  Furthermore  all  ukrin.iir.ei  is  collected  inohmcu-ih  No  one  im.  h-liiir. 
us  or  ;-«i  win  be  able  b  lie  specific  repl  tei  lo  in»  indi*  iliuL  project  or  or  pail  alien  All 
h  fee  mu  lion  collected  w  in  tie  held  in  strict  confidence  bn  the  SB  Suren  cliti  win  noi  be  released  lo 
anj  01  Irr  commercial  or  £o>eninenial  crj-ailu  ions  The  iijren  re-port -e.-;  ■'■■ill  tie  atahied  h;-  Ihe 
SE1  Qnh  a  pp repate r;ti ti sties,  es oer pis  ind  other  11  crt-allrbuuble  quotes  LnlraccsHe  to  imp  person, 
project  or  or  la  lion  '■'■ill  be  published 

Fleiieencounge  ilk-  rjetecied :  ur-e-  respondents  Loats"er  I  heirqitmliceiruiG: lascmdidJj  aid 
complete  b  is  lhe»  can.  The  reiUliw  ill  be  meful  lo  ion  u;  and  others  cub  lo  the  ojc  lenl  ltd  Ihe 
:ur?n  Eipoidenis  are  open  and  hones!  in  I  Ir  i  cpliex.  ISocako  Ihe  repotEd-  ae  inonr-cmous  there 
L  no  need  lo  ftde  wabneaei  orernbellish  .-irenr.'lis 

.1  Plei:c  i'A-mif-  appreprute  repondenlswilhinpacrp^ri^jiionbo  pmiepic  Hi  the  sir".;- 
Potential  re iponderfa  include  project  truhir.cn  program  miiupcr-  depJn  project  nu  rarer-  or 
depul*  proj  ran  iruiur.cn  fee  all  projects  wjUiinjour  or^  ail  alien  Both  prime  coniracls  md 
subcontracts  should  be  included  in  Ihe  pool  of  potential  repondenls  Please  don'L  'khenn  pick"  Ihe 
mosl .  ijovs-IijI  prcjecls  11 1  ccpiilb  in  fort  art  Lo  include  an  dkire.isecl  project;  ihil  mr-  es  L-l 
:  -iiiil.iit-  Ihe  pool  ofpoleriial  espendenb-;  should  include  all  project;  regurdlesr;  of  sle-conuud 
i;-pe  applicjlion  domain  wfElher  cr  not  Ihe  system  if- piece  den  ted  or  Ihe  muniril*  Isnelof  the 
■retail  jii.rialunillC'Whirh  Ihe  projects  rport 

We  wcrild  like  lo  receive  reiportit;  ln.rn  a;  iiiur-  projects  as  possible  I  fjcu  ac  unwilling  cr  unable 
lopno  kk-  responds  from  all  of  jour  projects  pleisc  selectn  niidcrn  simple  from  them  One 
method  of  ensuring  unurbia-ed  simple  tilocreaie  a  complete  list  ofall  projects  and  choose  ere ij 
"N“"  projecL  lr-rn  ilul  list  For  oc.uipk-  if  jou  decide  lo  peoo-ide  10  re-pon*;  from  a  total  of  I  ■■■ 
prcjecls  jouwouldchoorieei-eir  10“  project  (bMG).  Jf  joudeckt  Loponide  XrespcnKS  jou 
would  choose  e»ern  ^ i  projed  fb^flj 


I  li p.-u  I r -■  ill  -  ->ire;y  i  h  ihtiv  "rr--'+:-.i"  ind  1  "p F7y[ mi"  will  U  uk-J  i n u-r k ■  ry< a-- lv 
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4  Report  ill.-  mauler  ol  projects  chosen  io  ilk-  SEI  >!□  telephone  ji  41 2-263-41  32  orv  ia  email  .a 
je  Im  >i>  :-e  Lamiedii 

5  IH-iiihiie  ilk-  teller  •  Ehclosue  4)  as  well  a- ilk-  Non-I3:-elc«ir I  Thar-  Policy  i  l-.nck-c  un.-  21  ant 
ilk-  Definiliorei'Enclorsure  ?)  to  alhekcledsutey  responder!-  include  you  rquetl  or  dfrcikn.  hr 
ilk-in  In  ptj-iicipar  .aid  icbnUfy  ilk-  pmject^j  fee  wKch  ihe-  should  re.p->ni 

Please encourage  ilk- selected : ur>ey  re:f"Hi'k-iMM'.'.ii:--,vr'lp.-ir'.|'L-.ip.HiiLiiP.-.  icuididh  .aid 
complete  I-  as- the-  can  The  resulswinte  useful  lo  ;-ou  us  iirJ  others  only  ioilk-e.ck-iii  ilunlk- 
suney  r. pondenis  are  ■  fen  and  henesi  in  i  lr  ir  rpliei.  I  ■*.-  In  ilk-  monymihi  of  ilk-  response  ilk-n- 
L  no  need  In  I ■■>.-  weuknes-ses  oreinbellish  siren  lilts 

6  Please  ensue  ilui  sufficient  I  ine  w  ill  be  pro*  ided  for  ilk-  project  mutagen;  and  their  staff  In  oornpler 

ilk-  i  >ir  pee  lesis  hone  shown  that  completion  of  Ihe  ques liomaire  r-  pk1  ill;-  requbesp  to  In  60 

min  lie  i 

I  I  iriiir.  Ihe  lime  established  bn  cnllixt  responses  3E1  w  in  ccnlad  you  oryour  it-. if  no?  In  cheek  on 
the  .  un-ey  .aaiu.-  When  conucrd  h-  Ihe  SEI  pfease  check  -,-iih  you  selected  nespondert;  lo  .we  if 
they  Itr-  .aiMni  led  Ihe  t  rs-patsei  and  nepcrl  Ur  number  of  responses  siltniiied  in  Ihe  SEI  ■>  ia 
telephone  il  4 12-263-41 32  or  fla  emai  I  al  jeltikil  r-.-i  onuedii  Remember  lhal  Ihe  SEI  win  nol  knoo- 
which  speci  Ik-  projects  lu>e  been  :e  lecled  io  participle  in  I  lr  ittney  so  Ihe  3 13  camof  remind  Utem 
Looomplele  Uteirquea  imrricE  they  musi  r  h  upon  you  lo  do  Lhal 
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zmtxm  iikmin  Hfc.fr it  imiKUDi;v 


nil  £4k4K 

Ail  IP! PS  VtiEilZliai-XlU 
Ttft  iHEtKLMIM  ■  Fta  IMtyKWM 
Wrfe-r^u:  kgbAtawn. 


/.Ur  ViUM  iif'.'Ui'  flliTii.’.!iiul'.Uir.r 


J  J.l V  1?.  SWIS 


I  Jr_r  'i  m  >:  r  I  i  '  ■ ;  I 

Yinrr  in  in  ip'isi  'n?  r^s  seJawl  |iji  In  rankipiie  in  ii  rirvi-  ilui  is  fcetng  ocndicHd  by  SIILA  ‘ailTi  ihr 

nneipti nfn  iif  il'f  5i,ifi*"irr  Evayiiwuig  lwiwLs  ■  SEI>  Hid  ti  {rw-liniii-in  ^'iih  die  CHlhe  or  ihe 

I  li>ci'i>ri.-irl:i[Y  nT  Ikviti;  f-rr  ApVli*Hin>,  TwtnKjiJCJ-  Hid  1.0  US&  AT-ftlU.  Aleng  M  1 1 1 1  OHhmr 

pupil  ill  ill  i;yi  >.  limn  r> nliia.v..H  \  ilm-nrliiitj  llh-  iLTiu <r  iiKticiiv  your  replK5 -will  help  pplV'ldf 
Liuhlwmlhr.  :|u:iij  i. ilr.ii  nriilriKr  ibcul  Ilk-  «i|w  ^TyTklTO  enC'aWimE  Kli^iliil  imlw  KquIsKCO  Jfcj 
lluMikf-Hibiiil  :  if  ii  [mu  s  1  on  ■ .  :iu:li  ir.firiiEiliiir  1 1-.‘  r. -K.fnv  h:  i Ixm  sorely  "■■■'"j  ifld  dlt  resilis  0 i 
lius.  iuiYiy  II  ill  Ik  IiT  LiCkul  inpinluiLL  In-  lull  I  In  HhqurinH-nt  rj"  1  ;nij  ito  US  Litre  1*7  IIKhSliy. 

I'kapoonirpklei.ht  quadfonHR  n  canlO  i  UKlctMipkielydsianpttsIblyaih.  Tbtrtidii  v ill  lm 
nwful  w  jw,  us -ir..i  ndmr  mil  in  die  «i«u  ilui  you  Jiri  ihe  oilur  suiyey  rapaiikmi  mf  npca  ind 
NxmiI  in  3-niir  rnjlinr  HI">lA  hn  is  i  fad  ih-e  Safi^-jite  F:r^irti_v:rjni;  injure  nt  (5E3t  in  uvreuk:  tfcii  m  '.m 
SF.I  L-.  :miii:L  [i--ns-nl  Miy.ir-r.ilim  u-jlhnfl  dilWI  WJ  » inlwr  JCCflilCK  if  5r.ppIx.T--  Il  li  kilAnAffii 
TiEiiiJiiiiiir;;  :iinfi:L-ini.iliiy  i.if  iiil'iimuiiiir,  iyf  cn.pj  frijsn  Mi  jnsvumi-n  pidCnflll  jM  tfcltJUt 
miElrjLiAHY.  All  ir> -ii-.  will  be-  rnllrclni  iiunymnudy  hr  SCI;  i»-.  [rpr.  imludlFf  NI>1A  OT 1  C'il  WII 

•nm^oTisii,  will  Mf  In  epgtffk  rcfbns  lo  any  iitdhqdral.  prcgerc.  nr  aiGuduiiun  All  rtplkt  will 
It  "r  J  in  -^m-L  l--.w 1 1- ■  1  rur r  bySEI-  Hpy  wi^  bn  n«ewihfa  only  Inuntaiaed  iS!I  sii.1l.  jirJ  v-Hi  na  hr 
KKII  Irf  any  NIkt  .'.iiiiin-.il  r*  JU  urn  men  1*1  ■,TajnilW.K3flS  Tin  mulls  L.  Ill  hu  Ipubllshaij  r.ll  ,  Ji 
nys.n-'e-'in  iiMufcj  nrd  -twitM  in)  njhpr  hvi-mijhicNn  quotes.  There  ii-  ro  ivietl  m  hide 
i.i-  iirs  lii  i.  (Jnnqthi.  t.i«r  pn.:  lt»|j  turin  shnwn  (haJ  o;nvl«H.i«i  o*  die  quevcioniwilri;  i> fii-illi  injurci 
10  In  4^  in  -nilrE- 

T-nromplilc.  iIk  su^\;y,  omipleic  ih.1 1'nllnwii^  slips 
I.  hunryoir  browser  in 

1 1 1 1 1  ri  n  >j-i  L‘ — 1 1 . i" r  .1 1 k  n L1  Li->J  i_  t i r  h n  p  1 ':'.'nny_  1“ 'in’ll  JS;1 

u.-tKW  you  arill  ha  rwued  u  uniqpji  md  nndouily  gernruiid  liKJ..  rris  is  tTH.Ti  t^il ...  pknw 
iwW  H-jwn  “ill  need  k  later  u.fien  ofimpleim^  ihe  sinvy.  YOU  ft  UFLLillnwi  yiw  n>  rdnin 
j™3  pwapldn  jwr  qunakrTram?  rr'er  Meslnes  If  rU-££^3i4ry.  ^  Ii  ■/  iilii: J.ii'smi  v;s,ir 

UHV^nAiy. 

--  Ihiim  ji.nr  iiwr tn  YOUR  URL  obtained  >r  sup  I  in  jshi  ilu  sun  ej  ^imiumuau'c.  Vim 
uni  luiiu’iMkiiiriJ  |'r-m  iheYOUR  UkL  I  isprs'-enieiudimnihe  iui'.'iy  fiam  Eiicr  j  I1  a i 

<if  jLur  iiwri  rim.— i;  fur  Ih;  HnJ  qansaKO  KUKJ  TTo  pnHCCI  ^llll  OllXI^WliLy.  dm  nrl 


1 '} ’m h‘i >■  L ■  i  *  in  ’.'nll.’H.irjA  i'rs  i r  ’I Ti .  ' ¥-■ 1 


NATIONAL  DEFENSE  INDUSTRIAL  ASSOCIATION 


SOFTWARE  ENGINEERING  INSTITUTE  |  195 


2 


LlMivj  J  : ■ :■  ViUU  llul  ilk n'i[iL>  i: r  In:  li  al  lliu  ilnilil;.  CCT  >  LMmzH.  u  i  ir  |*a-. ^iuiiii.  :a  milt 

or^uilidLitMi.i.  Ihc-  user  iiimd  arid  (Nuwufll  j»  year  fetyt  u  «iii|kkiiu|  Uk  u  sty  cm 
r.mlupk- M.*»ii:hv  .jiil  'Mil  Jl>ii  F*Bi1d(  JtLtii  iu  the  rejun  oo  Birvej  icmJi>. 

}.  fuMi^cU:  d*  Mirvcy.  rikk  ah:  'SAVP.'  h*j  1 1  im  I  I  kjk  h  i  r  v.-i  L  WPe*  V'>J  Iwr*. :  EWipIlijcd 
:iiiJ  ‘.J'lTij  ill.1  s  u  '.u  .  ■_  i  >!  i 1 .1  1 . 4.*  j  I  ■  i  pr.  li :  r  |,vh  m-  r  I T  i  i  i  ■  ■■■ ;  I  ■ 1  .‘Hi  l.-.'/i  :n  ,■  J.r  L‘:i"iin.|r[J  ,~a  in[jf 
nktl  bmwSri  ■  and  Clkk  Ilk  ’Sul'  :iT  haJIQ4  In  MVLrclif  EnAamfl  :ld-  11  Ir  i I :  ■  ■  F . I 

Aa  -s:l.-il  :k:  ■f:h->.;i:nn  lin-  Jim  will  'asj-  lilt-  tun:  nwr  naiif  n|ll  in  iccysi  |ii  ijil 

mi . tty  ifikbI  stfU-d:  Mm  ry  irejilk  wlicn  -if  fcwania  sm*hbV  Pjf  Ul  k'SJl  J  full  C1UT,  fa  Will  b« 

nmltMUjlitiiuailj  In  .-i:^  vim  full;. Li:nip!i:l.-  j  .nin  rv  qnr«lii:iT-:,nT  li  wi  II  pr'.">l':- n  I?  J  wllllj  jpnlTrt 

w  Ik.  Ii  i  ml  lie  .  I'li^ji*.1  ilii  ptuVi  imam  4 if  i  iiii  ptju  nrjl  ■rr.ir  >  nl.ir.  v.  ■?,  tHlyij  ir  lilt  ll  Ju-jji: 

Hi;  w*  silM  Ml  l>:  jclf  e  and  j'jibhk:  |nr  i;,-l  nl  ‘jJ.i^  jIi^i  uP  mi  iil  i1ii  :h  -,..  '.  rn’a&M- 

July-KiXilhrai^  l'!-Sci|Twmlvr.>iXi  ivviw  ropond promptly  Wplibin erv a- di.ii.--  ':':u  : u ■  ■  j l-_ ■  i a: rj 
wj!|  ciki.ci  ‘.thi  fnirp  iim<  i link  isi  ncmiraJ  V'.'dj  yhuut  'li;  imjmLirEa  of  this  suray. 

I  haul-  you  for  puiidiuiiitg  Indih.  naairch.  'Ilk  :t-iil:-i:u-.  itflufra  Mr-  ■_ tin i i ■  ■  -  ni'  i.-jimr 
crainEcriru  pmctioc.  a;  wall  n  gc*m  mien)  aiqtuJiJan  besi  pucikxi  And  yutr  own  deatfc^wni  a  ink 

JfiiKcnefy. 


1  jAirn:  i-  ?  Fin  HI.  Jr, 
l.imlEiiinl  1  i.-iK-nii  :  C  'A-V.P  ,„k.'i  i 
PinJd=nl  £  CED 

Miliuul  IlfLidE  &iilu>Jrijl  iltj I i I m 
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itumb  !  .-juaaa  ■niwiiv. 


Enclosure-  5:  Terms  : mi  cl  I  Minim  ns 
LUMliiCL 

Aflam  pr]'EO.:;]LA\i-  H  an  Hlepaled  collection  of  efi'ctlp  b™  a  wtnnbior  te  ain  lhaL  prockice-; 
dclh^rabk::.  f  mt . i.-r  under  a  ;p=cific  fflBlBiM  SfciiiHiflfl 

*  Ther«tiiL*ii  k-'in  L:  .in  '-r'-.iiiLzai  f. 4i  il  '.-iiLir-  that  mr-  tv  ineohiinr.  pn.jeci  ai  iniegrjed 
product  team  ■ I  PTi  a  program  i  -Ji'>  L  kti  or  an  a.  ioiiihh-  ol  member  iu.iirio.-d  In.tn  more  ikiii 
one  "ir  iiiL.iiM'iul  mil  "I Til-  ooniraclor  n-.iin  Ids  r-.  poitihiliF-  for  ihe  de*ebpmerl  or 

.  uslainmenl  ofaea  more  de  Iferables.  Id  a  ciftomei: 

*  1  vll'  vr-'l-k-:-  Mi_r-  be  produce;  (e  g  if.aern.  :  1 1 1-. ; re itl -component;  daluiari:L‘cc.er>i:e:- 

*  Acvslomer  inr-  be  Ihe  gn-emineni  another  com  me  re  Li I  cccdniiuiion  met  her  division  wilhh 
joLrcr|vuriuUon  anctlErcoriradcr  loam  *dLhiir*crjrorganiiLj  ion.  eb 

*  A  com  rin'nul  ■■MIx-n  ion  is  a  comninnenl  Id  perfonn  a  defined  last:  in  relirn  br  defined 
com perta lion.  Ccmraclual  obligation:  are  characterized  bj 

o  a  de  fined  scope  f  ■■■.■  ■'■  rt. 
a  a  de  fined  period  of  performance 
o  a  de  fined  cod 

a  ara^nmerl  Loa  defhcdccreculinx  organization 

o  Kilned  reipcmdbilift  Ibr  porter  maoe  b  a  member  of  ihe  cortrad  kig  team 

>.  '.h 1 1 r  p.  i ■  l 1 1 J -a i :'-ij i i O' rL  iii  r-  be  defined*  i a  con Irdtiriwiiiuhe  ^o^rnnenbcoriracIswiLh  other 
comnerc ial  organizations.  wcrkagreemert3'»,ilhki  aiicrtjniz  iiicn  eLc 

Topical  Characicrisljcsofu  [rojecl  include 

*  Aligned  projecl  manager 
■*  IJe  fined  so  tad  Jo 

*  Defined  budget 

A  Work  breakdown  suuciue  fWBS) 

■*  Cool  acootiri  management 
trample;  of  PROJECTS  include  ihe  I  oik  inf 
+  ConiRn'kv  ABC  is-  under  contracl  ficm  Ihe  Je>eminenl  b  delfrer  Ihe  F-ira  aircraft 
■*  (.  Oflinn'kf  DEP  L  under  ccniracL  iTomccriuiirior  ABC  lo  delh-erengires  for  lie  Ken  aircraft 

*  The  A*  ionics  I  ii>  i:  Jon  of  Conlradcr  ABC  is  asked  Itj  Ihe  Krai  program  lo  [rotide  the 

ii  jrifuiion  aid  cormimicjiiore  a*  ionics  for  Ihe  P-ira  ThL  e  Hon  lias  a  defined  rfcope  budget 
and  schedile  negotiated  tel^een  the  F-iii  FK1  and  ItaA*  ionics  Dhisionmd  document'd  in  a 
Work  Package  lie  sn  i  ft  ion  a  nd  Memorandum  of  A  giee  me  h  The  Avionics  I'h  Li,  ti  ha;  signed 


'  Tin  ■  mu  "praiitl"  inJ  "pror rill"  ir-:  u^d  in  Klinn  3 N  .  Ilim^kil  Itii  jiny 
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■TliM,*  rfi&  JHIdl  uFlUfttii*.'. 


a  iT'T.rjin  M.ii.ifvr  for  Ihs  el'fcrl;  TheA-t  p.4iic!-  IDt'i  i>p.4i  FkisjWBS  fcr  and  uncles  oosisis 
NKtvi  For  ik-  war  k 

The  FrJInwlni;  i:-  tm  iiu-r  ample  cF  a  PROJECT 

*  Afeamcf  :ic  engineers  and  otesir^vrs  wilhin  CoitricIcrABC  is  lasted  !o  de*c  top  die  wiring 

li.fi>.-.:.:  Ur  ik-  l--.t.t.c  lircrafi  'Ik*  learn  rpoi&loon?  of  ik  cnrjiLvrihr.  bail  ivp*n ii*  ttihe 
F-ira  PM  The  w  rirc^  lunr.-:  :•  is  an  element  cF  ik-  h-m  and  cods  For  the  de>  el  opine  nL 

effort;  jf  collected  wiihh  ik-  l-r.r.r  jo:  o  Lining  saslem. 

A  CQPfTTLYCTQE  ORGAN  IZATlCrN  is  ill  administrate  im.  nr-  wilhin  which  ■  pa  sibh  irum- 1 
l^ojecls  or  FfnilT  won-,  effort;  je  opguntod  mder  common  imiLiry  nrnl  uid  policies. 

*  'When  Lh  in  king  ihoii our  cceiiraclor  opguniiiUoii  plea-e  aitwer  For  Ihe  unit  Lo  which  due 
coniraclor  team  report;  actnint-dnlhelf  e.g,.  a  sle,  df-  Lkn  or  ok.-pimir.-ni  nol  Fora  I.  river 
en-.-q.ri  e  oFwhich  the  co  line  For  ons-a  nil  .r  ion  n*r-  be  j  pent 

*  Oe  pend  fig  on  thesis:  and  scape  oFlIe  prcjecL  ik-  c:rim.j:i'jf  crguriiiaLion.iidilr  contractor 
■■.--■in  iii.i;-  been?  ;rd  i If.-  .  .■  iil-  jiid.gr  iIf.-  .  ,iin-  contractor  organization  mr-  be  reipcnisihle  For 
iiioe  ill. ii  one  project 

■*  The  contractor  ctginizci  ion  in.r-  be  Ute  prime  contractor  or  ■  :u  been  tractor  1L  mi;-  cr  inr-  noL 
ha*e  HjbccnlnKLnracrf  ifccwn. 
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APPENDIX  D  Response  Distributions  for  Individual  Survey 

Questions 


Per  the  terms  of  agreement  with  the  survey  respondents,  the  contents  of  the  Body  of  the  Report 
and  Appendices  A  through  C  may  be  released  publicly,  subject  to  the  copyright. 

The  contents  of  Appendix  D  may  be  released  only  to  OSD,  NDIA  Management,  and  respondents 
to  this  survey  until  November  20,  2008.  After  said  date,  these  restrictions  are  lifted,  and  Appendix 
D  may  be  released  publicly,  subject  to  the  copyright. 
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